Nov 02 06:13:38.423364 ip-10-0-142-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 06:13:38.423376 ip-10-0-142-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 06:13:38.423385 ip-10-0-142-103 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 06:13:38.423697 ip-10-0-142-103 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 06:13:48.522330 ip-10-0-142-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 06:13:48.522349 ip-10-0-142-103 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 623e4f3a5e794199a275c899378c54a4 -- Nov 02 06:16:09.247328 ip-10-0-142-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:16:09.693391 ip-10-0-142-103 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:09.693391 ip-10-0-142-103 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:16:09.693391 ip-10-0-142-103 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:09.693391 ip-10-0-142-103 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:09.697837 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697759 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.697837 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697837 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.697837 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697841 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697844 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697847 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697850 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697856 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697859 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697862 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697864 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697867 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697872 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697875 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697878 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697880 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697883 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697886 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697888 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697891 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697893 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.697930 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697904 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697907 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697910 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697912 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697915 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697917 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697920 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697923 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697925 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697928 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697930 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697933 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697936 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697940 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697944 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697949 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697953 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697956 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697959 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.698406 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697961 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697964 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697966 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697969 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697971 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697974 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697976 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697979 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697981 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697985 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697987 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697990 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697993 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697995 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.697999 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698002 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698005 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698008 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698010 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698013 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.698858 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698016 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698018 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698021 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698023 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698026 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698029 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698031 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698051 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698056 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698060 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698064 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698067 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698069 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698074 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698077 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698080 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698082 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698085 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698088 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.699337 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698090 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698093 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698096 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698099 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698102 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698104 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698107 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698111 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698115 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698117 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698120 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698123 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698126 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698129 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698131 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698134 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698137 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698139 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698142 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.699825 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698145 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698148 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698153 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698156 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698159 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698162 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698909 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698916 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698921 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698925 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698929 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698932 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698935 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698938 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698940 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698943 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698946 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698948 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698951 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.700292 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698954 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698957 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698964 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698967 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698969 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698972 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698975 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698977 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698980 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698983 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698985 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698988 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698991 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698994 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698996 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.698999 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699002 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699005 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699007 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.700768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699010 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699013 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699015 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699018 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699021 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699024 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699027 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699029 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699046 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699049 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699052 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699056 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699059 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699062 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699066 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699069 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699072 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699075 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699078 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.701253 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699080 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699083 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699086 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699088 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699091 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699094 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699098 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699102 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699105 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699108 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699111 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699114 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699116 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699119 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699122 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699124 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699127 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699129 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699132 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699135 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.701692 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699137 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699141 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699143 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699146 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699148 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699153 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699156 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699158 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699162 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699164 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699167 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699170 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699172 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699177 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699180 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699183 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699185 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.702183 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699188 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699191 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699193 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699196 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699199 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699202 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699204 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699207 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699214 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699216 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699219 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699222 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699225 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699227 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699703 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699708 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699710 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699713 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.702648 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699716 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699718 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699723 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699726 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699729 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699733 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699738 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699741 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699743 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699746 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699753 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699756 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699759 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699761 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699764 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699767 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699770 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699772 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.703113 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699775 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699777 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699780 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699783 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699785 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699793 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699804 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699808 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699810 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699813 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699816 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699818 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699821 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699824 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699827 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699830 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699833 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699836 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699839 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699842 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.703566 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699844 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699847 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699849 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699852 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699855 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699858 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699861 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699863 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699867 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699869 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699873 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699876 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699879 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699882 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699884 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699887 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699889 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699893 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699896 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.704102 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699899 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699901 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699904 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699906 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699909 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699912 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699914 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699916 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699919 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699921 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699924 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699927 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699930 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699933 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699935 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699938 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699941 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699944 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699948 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.704558 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699950 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699953 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699956 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699958 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699961 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699963 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699966 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699969 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699971 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699973 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699976 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699980 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699982 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699985 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699988 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699990 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699993 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699995 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.699998 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700000 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.705005 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700003 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700005 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700008 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700010 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.700013 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701160 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701176 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701185 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701190 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701194 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701198 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701206 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701211 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701214 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701217 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701221 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701224 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701227 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701230 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701233 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701236 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701239 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701242 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701247 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:16:09.705524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701250 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701254 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701257 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701261 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701265 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701268 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701271 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701275 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701278 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701281 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701284 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701287 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701290 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701295 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701298 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701301 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701304 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701307 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701310 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701315 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701318 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701321 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701324 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701327 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:16:09.706090 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701331 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701334 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701337 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701340 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701343 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701346 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701349 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701354 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701357 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701360 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701363 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701367 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701370 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701373 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701376 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701380 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701383 2579 flags.go:64] FLAG: --help="false" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701386 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701389 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701392 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701395 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701400 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701404 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:16:09.706676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701407 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701410 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701413 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701416 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701419 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701422 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701425 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701428 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701431 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701435 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701438 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701441 2579 flags.go:64] FLAG: --lock-file="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701443 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701446 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701449 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701455 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701458 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701461 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701464 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701467 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701470 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701473 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701476 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701481 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701484 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:16:09.707338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701487 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701490 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701493 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701497 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701499 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701504 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701507 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701510 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701518 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701521 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701524 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701528 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701531 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701534 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701537 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701540 2579 flags.go:64] FLAG: --port="10250" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701543 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701546 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01b93421fa03ee9b5" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701549 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701552 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701555 2579 flags.go:64] FLAG: --register-node="true" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701558 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701562 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701565 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701568 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701571 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:16:09.708055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701575 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701578 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701581 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701584 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701587 2579 flags.go:64] FLAG: --runonce="false" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701590 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701593 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701596 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701599 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701602 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701605 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701609 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701612 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701615 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701618 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701622 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701625 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701628 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701631 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701634 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701639 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701642 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701645 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701649 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701652 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:16:09.709153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701655 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701658 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701661 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701664 2579 flags.go:64] FLAG: --v="2" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701669 2579 flags.go:64] FLAG: --version="false" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701673 2579 flags.go:64] FLAG: --vmodule="" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701677 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.701681 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701823 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701828 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701831 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701834 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701836 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701839 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701841 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701844 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701847 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701849 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701856 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701859 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.710239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701862 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701865 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701868 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701871 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701873 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701876 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701879 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701881 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701884 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701887 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701889 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701892 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701895 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701897 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701900 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701905 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701907 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701910 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701913 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.710836 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701915 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701918 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701921 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701923 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701926 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701929 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701932 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701934 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701937 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701940 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701944 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701946 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701949 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701952 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701954 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701957 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701962 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701966 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701969 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.711329 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701972 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701976 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701979 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701982 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701984 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701987 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701990 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701992 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701995 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.701999 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702001 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702004 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702007 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702009 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702012 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702015 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702018 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702021 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702026 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.711812 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702029 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702045 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702050 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702056 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702061 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702065 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702068 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702071 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702073 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702076 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702080 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702083 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702085 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702088 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702091 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702094 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702097 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702099 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702102 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702105 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.712309 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702107 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702111 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702114 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702118 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702122 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702125 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702128 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702131 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702134 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702139 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702142 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702145 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702147 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702150 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.702178 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:09.712786 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702212 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702216 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702222 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702225 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702228 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702230 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702233 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702236 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702239 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702242 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702245 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702248 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702251 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702253 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702256 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702259 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702263 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702266 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.713188 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702268 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702271 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702274 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702276 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702279 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702282 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702284 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702287 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702289 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702292 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702295 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702297 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702300 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702303 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702306 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702309 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702311 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702314 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702316 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702319 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.713637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702322 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702326 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702330 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702333 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702336 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702338 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702341 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702344 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702346 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702349 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702353 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702355 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702358 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702360 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702363 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702365 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702368 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702371 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702374 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702377 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.714112 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702379 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702382 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702384 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702387 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702389 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702392 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702395 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702398 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702401 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702403 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702406 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702408 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702411 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702414 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702420 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702423 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702425 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702428 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702431 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702433 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.714583 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702436 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702438 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702443 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702445 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702448 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702451 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702453 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702457 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702461 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702464 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702467 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702469 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702473 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702475 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702479 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702482 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702484 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702487 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702490 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.715050 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702492 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702495 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702497 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702500 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702502 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702505 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702507 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.702510 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.711127 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.711143 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.711780 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.711787 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711859 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711865 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711869 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711872 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711875 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711878 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711884 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.715547 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711887 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711890 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711894 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711897 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711899 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711902 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711905 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711909 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711912 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711914 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711917 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711920 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711923 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711926 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711929 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711931 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711936 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711940 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711943 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.716023 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711946 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711953 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711956 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711959 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711962 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711965 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711968 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711971 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711973 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711976 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711979 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711981 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711984 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711987 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711989 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711992 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711995 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.711998 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.716494 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712008 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712011 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712013 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712016 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712019 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712021 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712024 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712027 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712030 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712047 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712050 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712053 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712056 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712059 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712062 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712066 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712069 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712072 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712077 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.716924 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712079 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712082 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712085 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712087 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712090 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712092 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712095 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712097 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712100 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712103 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712105 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712108 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712110 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712119 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712122 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712125 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712128 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712130 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712133 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712136 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.717415 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712139 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712141 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712144 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712147 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712150 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712152 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712155 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712158 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712160 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712163 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712166 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712171 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712173 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712176 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712180 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712184 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712187 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712189 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.717887 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712192 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.712220 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712360 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712365 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712368 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712371 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712373 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712382 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712385 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712388 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712390 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712393 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712396 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712399 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712402 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712404 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.718352 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712407 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712410 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712412 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712415 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712418 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712421 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712423 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712426 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712429 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712432 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712434 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712437 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712440 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712442 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712445 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712447 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712450 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712452 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712455 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.718726 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712459 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712462 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712465 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712469 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712471 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712479 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712482 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712485 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712488 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712490 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712493 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712495 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712498 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712501 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712503 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712506 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712509 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712512 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712514 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712517 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.719283 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712519 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712522 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712525 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712527 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712530 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712532 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712535 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712537 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712540 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712544 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712547 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712551 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712554 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712557 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712560 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712563 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712566 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712569 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712577 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.719768 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712580 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712583 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712585 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712588 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712591 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712594 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712596 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712599 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712601 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712604 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712607 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712609 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712612 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712614 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712617 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712620 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712622 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712625 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712627 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712630 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.720247 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712633 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712635 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712640 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712643 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712645 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712648 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712650 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712653 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712655 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712658 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712661 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712669 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712672 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.712694 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712731 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:09.720714 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712735 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712738 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712740 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712745 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712748 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712751 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712753 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712756 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712759 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712763 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712767 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712771 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712774 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712777 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712780 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712782 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712785 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712788 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712791 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:09.721118 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712794 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712797 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712800 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712803 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712806 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712808 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712813 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712816 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712819 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712827 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712830 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712832 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712835 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712838 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712840 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712843 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712846 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712849 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712851 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:09.721579 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712854 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712856 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712859 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712861 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712864 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712866 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712869 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712872 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712874 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712876 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712879 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712882 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712884 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712887 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712890 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712893 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712896 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712898 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712901 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:09.722131 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712903 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712906 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712908 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712911 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712924 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712927 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712929 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712932 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712935 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712937 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712940 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712942 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712945 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712948 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712951 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712953 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712956 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712959 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712961 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:09.722573 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712964 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712966 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712969 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712972 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712974 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712977 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712979 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712983 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712987 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712990 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712994 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.712997 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713000 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713003 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713005 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713008 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713010 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713013 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713021 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713024 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:09.723017 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713027 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713029 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713044 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713048 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713052 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713055 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713057 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:09.713060 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.713116 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.717871 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.718817 2579 server.go:1014] "Starting client certificate rotation" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.718909 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.719906 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.720783 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:16:09.723529 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.721923 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:16:09.735145 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.735124 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:16:09.738628 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.738613 2579 log.go:25] "Validated CRI v1 image API" Nov 02 06:16:09.739452 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.739418 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:16:09.743438 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.743421 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 805c9131-05aa-4245-b502-f7e0bce13636:/dev/nvme0n1p3 c9e024ef-8c1d-40c8-bee7-12f99b722eb0:/dev/nvme0n1p4] Nov 02 06:16:09.743486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.743441 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 06:16:09.748995 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.748893 2579 manager.go:217] Machine: {Timestamp:2026-11-02 06:16:09.748421846 +0000 UTC m=+0.377939272 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2244606ee1de3a7cd73f53ebb7c48b SystemUUID:ec224460-6ee1-de3a-7cd7-3f53ebb7c48b BootID:623e4f3a-5e79-4199-a275-c899378c54a4 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:df:5e:c8:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:df:5e:c8:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:27:74:b7:fc:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:16:09.749509 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.749499 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:16:09.749586 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.749574 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:16:09.753974 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.753954 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:16:09.754110 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.753976 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-103.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:16:09.754151 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.754118 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:16:09.754151 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.754126 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:16:09.754208 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.754159 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:16:09.755066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.755056 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:16:09.755901 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.755892 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:16:09.758938 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.758926 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:16:09.759077 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759068 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:16:09.759839 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759830 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:16:09.759870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759842 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:16:09.759870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759852 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:16:09.759870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759859 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:16:09.759870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.759867 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:16:09.760306 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.760285 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:16:09.765015 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.764995 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:16:09.769417 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.769404 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:16:09.770267 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770255 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:16:09.770324 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770285 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:16:09.770429 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770419 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:16:09.770464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770434 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:16:09.770464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770440 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:16:09.770464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770445 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:16:09.770464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770451 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:16:09.770464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.770456 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:16:09.771427 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771418 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:16:09.771468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771428 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:16:09.771468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771437 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:16:09.771468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771443 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:16:09.771468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771452 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:16:09.771468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.771461 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:16:09.773018 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.773008 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:16:09.773018 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.773018 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:16:09.776766 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.776752 2579 server.go:1263] "Started kubelet" Nov 02 06:16:09.776920 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.776892 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:16:09.776978 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.776878 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:16:09.777024 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.776989 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:16:09.777482 ip-10-0-142-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:16:09.778605 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.778588 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:16:09.779602 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.779588 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:16:09.782782 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.782763 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 06:16:09.784389 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.784368 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:16:09.784932 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.784916 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:16:09.785050 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.785015 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" Nov 02 06:16:09.785757 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785741 2579 factory.go:55] Registering systemd factory Nov 02 06:16:09.785757 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785756 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:16:09.785887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785779 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:16:09.785887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785782 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:16:09.785887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785803 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:16:09.785887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785857 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:16:09.785887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785867 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:16:09.786026 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785936 2579 factory.go:153] Registering CRI-O factory Nov 02 06:16:09.786026 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785946 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 06:16:09.786026 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.785994 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:16:09.786026 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.786008 2579 factory.go:103] Registering Raw factory Nov 02 06:16:09.786026 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.786022 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 06:16:09.787020 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.787005 2579 manager.go:319] Starting recovery of all containers Nov 02 06:16:09.800424 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.800395 2579 manager.go:324] Recovery completed Nov 02 06:16:09.805293 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.805277 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:09.807656 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.807638 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:09.807724 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.807674 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:09.807724 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.807689 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:09.809029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.809015 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:16:09.809029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.809025 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:09.809029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.809027 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:16:09.809198 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.809076 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:16:09.810601 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.810582 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:09.810691 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.810616 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:09.810691 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.810630 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:09.811840 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.811827 2579 policy_none.go:50] "Start" Nov 02 06:16:09.811889 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.811843 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:16:09.811889 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.811853 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:16:09.813120 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.813104 2579 policy_none.go:44] "Start" Nov 02 06:16:09.852589 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.852564 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:16:09.854838 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.854825 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 06:16:09.854902 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.854857 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:16:09.854902 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.854867 2579 server.go:87] "Starting device plugin registration server" Nov 02 06:16:09.855128 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.855107 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:16:09.855173 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.855123 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:16:09.855213 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.855193 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:16:09.855295 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.855282 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:16:09.855295 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.855294 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:16:09.855774 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.855754 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:16:09.855854 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.855813 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" Nov 02 06:16:09.909737 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.909704 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-103.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 06:16:09.911513 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.909705 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-103.us-west-2.compute.internal.18e425d9fc9f45bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-103.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-103.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:16:09.776727483 +0000 UTC m=+0.406244910,LastTimestamp:2026-11-02 06:16:09.776727483 +0000 UTC m=+0.406244910,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-103.us-west-2.compute.internal,}" Nov 02 06:16:09.927960 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.927935 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:16:09.929071 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.929055 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:16:09.929071 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.929072 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:16:09.929200 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.929094 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:16:09.929200 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:09.929128 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 06:16:09.955543 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.955499 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:09.955646 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.955629 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-103.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 06:16:09.956244 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.956228 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:09.956308 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.956256 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:09.956308 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.956267 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:09.956308 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.956289 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:09.971988 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.971866 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:16:09.972072 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:09.971948 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ggwqs" Nov 02 06:16:10.029542 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.029518 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal"] Nov 02 06:16:10.029632 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.029582 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:10.030994 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.030978 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:10.031069 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.031006 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:10.031069 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.031021 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:10.032587 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.032576 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:10.032734 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.032719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.032793 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.032752 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:10.033287 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033273 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:10.033351 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033284 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:10.033351 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033299 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:10.033351 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033309 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:10.033447 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033309 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:10.033479 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.033449 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:10.034567 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.034550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.034654 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.034572 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:10.035315 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.035299 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:10.035388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.035323 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:10.035388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.035334 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:10.059015 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.058998 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:10.061639 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.061624 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:10.079713 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:10.079699 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.082660 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:10.082643 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.087584 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.087558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.087683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.087598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.087683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.087622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.153584 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.153560 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:10.155514 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.155495 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:10.156172 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.156151 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.185471 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.185455 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188712 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188791 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188791 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188791 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.188907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.188817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.278842 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.278796 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:10.278906 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.278882 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.378974 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.378959 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:10.383112 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.383100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:10.386269 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:10.386255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:16:12.099614 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.099550 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 06:16:12.100133 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.099677 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:16:12.100133 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.099679 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:16:12.100133 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.099695 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:16:12.100133 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.099700 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af4ae981bd91148c598c3a76754e16e4-db7866749802e67b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-103.us-west-2.compute.internal.18e425d9fe7735cc\": read tcp 10.0.142.103:58436->3.140.45.243:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-103.us-west-2.compute.internal.18e425d9fe7735cc default 5389 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-103.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-103.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-103.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:16:09 +0000 UTC,LastTimestamp:2026-11-02 06:16:10.030992645 +0000 UTC m=+0.660510071,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-103.us-west-2.compute.internal,}" Nov 02 06:16:12.141301 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.141284 2579 apiserver.go:52] "Watching apiserver" Nov 02 06:16:12.166040 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.166022 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 06:16:12.358593 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.358527 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:11:11 +0000 UTC" deadline="2028-08-15 08:16:52.573808424 +0000 UTC" Nov 02 06:16:12.358593 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.358558 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15650h0m40.215253358s" Nov 02 06:16:12.384676 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.384655 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:16:12.384937 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.384922 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb","openshift-cluster-node-tuning-operator/tuned-gtvjg","openshift-multus/multus-additional-cni-plugins-88wtj","openshift-multus/network-metrics-daemon-s4zwp","kube-system/global-pull-secret-syncer-h5h5g","kube-system/konnectivity-agent-gqnpb","openshift-image-registry/node-ca-v9gfs","kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal","openshift-dns/node-resolver-kcl2z","openshift-network-operator/iptables-alerter-7lsf6","openshift-ovn-kubernetes/ovnkube-node-bzgdv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal","openshift-multus/multus-29z9j","openshift-network-diagnostics/network-check-target-vd4bl"] Nov 02 06:16:12.386646 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.386630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.388276 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.388257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.389905 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.389890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.391221 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.391207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.391297 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.391281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:12.392650 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.392638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.392708 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.392694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:12.394283 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.394271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.395766 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.395755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.397054 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.397041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.398479 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.398466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.400137 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.400122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.402771 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.402752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.404075 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.404059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:12.404137 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.404121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:12.440016 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.439975 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:16:12.440117 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.440038 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-64trj" Nov 02 06:16:12.440117 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.440084 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:16:12.440245 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.440119 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:16:12.440245 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.440125 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:16:12.440245 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.440216 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:16:12.442555 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.442539 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:16:12.442637 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.442586 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:16:12.443799 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.443781 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:16:12.467595 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.467577 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:16:12.474418 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.474503 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.474503 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.474503 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.474655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtbc\" (UniqueName: \"kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv2m\" (UniqueName: \"kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.474872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.474958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwx9\" (UniqueName: \"kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.475266 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xmh\" (UniqueName: \"kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trqwx\" (UniqueName: \"kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.475726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.475960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.476228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdlv2\" (UniqueName: \"kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpxt\" (UniqueName: \"kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fg8\" (UniqueName: \"kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkbd\" (UniqueName: \"kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.476683 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.476392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.493249 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493231 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.493344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493249 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:16:12.493344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493263 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:16:12.493344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493277 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.493344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493300 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.493505 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493488 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.493554 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493546 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:16:12.493590 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493568 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.493590 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493576 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.493661 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493596 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:16:12.493661 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493624 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:16:12.493969 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.493951 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494000 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494015 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494017 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494033 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494068 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:16:12.494086 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494069 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494087 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494125 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494146 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494162 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494178 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.494338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.494248 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:16:12.540634 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.540613 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:16:12.540792 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.540774 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:16:12.540883 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.540842 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:16:12.540883 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.540858 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:16:12.541045 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.541028 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:16:12.576771 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.576855 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.576855 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.576855 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577014 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.577014 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.577014 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.577014 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwx9\" (UniqueName: \"kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.577014 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.576973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.577169 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.577195 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577241 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577261 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.577245 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.077204598 +0000 UTC m=+2.325769147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.577287 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.077275537 +0000 UTC m=+2.325840066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577406 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577567 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.577948 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xmh\" (UniqueName: \"kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqwx\" (UniqueName: \"kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577714 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.578704 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.577979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578086 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578630 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.579416 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578694 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.578718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlv2\" (UniqueName: \"kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpxt\" (UniqueName: \"kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579286 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.580138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fg8\" (UniqueName: \"kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579605 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579733 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579766 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579795 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579876 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581055 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579932 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.579956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkbd\" (UniqueName: \"kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.581907 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtbc\" (UniqueName: \"kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv2m\" (UniqueName: \"kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.580897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.581268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.581333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.581636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582281 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582581 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.582968 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582922 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.582951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.583856 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.583301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.584447 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.584423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.585273 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.585254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.628894 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.628848 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:12.628894 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.628868 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:12.628894 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.628880 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:12.629086 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:12.628934 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.128918328 +0000 UTC m=+2.377482876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:12.630633 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.630611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwx9\" (UniqueName: \"kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.631347 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.631320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xmh\" (UniqueName: \"kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.632278 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.632255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlv2\" (UniqueName: \"kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.632974 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.632953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpxt\" (UniqueName: \"kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.633147 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.633128 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fg8\" (UniqueName: \"kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.633217 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.633184 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqwx\" (UniqueName: \"kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:12.633629 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.633608 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkbd\" (UniqueName: \"kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.633822 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.633808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtbc\" (UniqueName: \"kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.636070 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.636052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv2m\" (UniqueName: \"kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.712139 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.712120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:16:12.724620 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.724599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:16:12.744276 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.744257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:16:12.752137 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.752122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:12.753472 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.753457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:16:12.758547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.758530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:16:12.776196 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.776182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:16:12.813281 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.813264 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29z9j" Nov 02 06:16:12.816528 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.816501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:12.940190 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:12.940154 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6c06b094cb0b06c560a952f3f0bc9d.slice/crio-18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73 WatchSource:0}: Error finding container 18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73: Status 404 returned error can't find the container with id 18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73 Nov 02 06:16:12.940561 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:12.940542 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05718cfa86289f66de168b803f5f9362.slice/crio-45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53 WatchSource:0}: Error finding container 45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53: Status 404 returned error can't find the container with id 45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53 Nov 02 06:16:12.945475 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:12.945462 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:16:13.062046 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.062023 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:13.084137 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.084117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:13.084254 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.084143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:13.084311 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.084278 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:13.084368 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.084340 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:14.084325245 +0000 UTC m=+3.332889773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:13.084368 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.084349 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:13.084461 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.084405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:14.084388745 +0000 UTC m=+3.332953293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:13.130854 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.130833 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:13.184832 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.184778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:13.184915 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.184907 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:13.184951 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.184921 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:13.184951 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.184929 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:13.185035 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:13.184974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:14.184962288 +0000 UTC m=+3.433526817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:13.314427 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.314383 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerStarted","Data":"18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73"} Nov 02 06:16:13.315392 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.315372 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"05718cfa86289f66de168b803f5f9362","Type":"ContainerStarted","Data":"45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53"} Nov 02 06:16:13.349423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.349406 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:13.441884 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.441783 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:11:12 +0000 UTC" deadline="2028-05-09 11:29:24.658233162 +0000 UTC" Nov 02 06:16:13.441884 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:13.441831 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13301h13m11.216404905s" Nov 02 06:16:13.565687 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.565666 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9ab1b6_621a_430c_9669_1071af47a675.slice/crio-87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f WatchSource:0}: Error finding container 87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f: Status 404 returned error can't find the container with id 87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f Nov 02 06:16:13.582911 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.582891 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd659865_edfc_4bb6_9c1c_38e02c8fe1e4.slice/crio-dddb6a1eea2cb25027d80dd2683e2fdf8565dd5a1fba441612510e13070fd835 WatchSource:0}: Error finding container dddb6a1eea2cb25027d80dd2683e2fdf8565dd5a1fba441612510e13070fd835: Status 404 returned error can't find the container with id dddb6a1eea2cb25027d80dd2683e2fdf8565dd5a1fba441612510e13070fd835 Nov 02 06:16:13.619428 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.619399 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd25d9ad_68e0_49c8_a9cf_0ca18a976680.slice/crio-73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902 WatchSource:0}: Error finding container 73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902: Status 404 returned error can't find the container with id 73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902 Nov 02 06:16:13.633481 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.633454 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853c435f_0786_450f_bdd9_f95cc551ba78.slice/crio-05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c WatchSource:0}: Error finding container 05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c: Status 404 returned error can't find the container with id 05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c Nov 02 06:16:13.641236 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.641204 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abb23a4_08db_4159_8715_df445adc7a71.slice/crio-e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4 WatchSource:0}: Error finding container e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4: Status 404 returned error can't find the container with id e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4 Nov 02 06:16:13.641775 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.641754 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb289fe_8e1a_4abe_a37b_4cc6ac343150.slice/crio-c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93 WatchSource:0}: Error finding container c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93: Status 404 returned error can't find the container with id c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93 Nov 02 06:16:13.644853 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.644836 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8c4326_8500_4a72_848f_7281b21b4853.slice/crio-e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb WatchSource:0}: Error finding container e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb: Status 404 returned error can't find the container with id e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb Nov 02 06:16:13.683280 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.683256 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa211aa_bb54_4375_bb65_793ae2e67721.slice/crio-497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b WatchSource:0}: Error finding container 497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b: Status 404 returned error can't find the container with id 497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b Nov 02 06:16:13.694493 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:13.694474 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37012b0f_1510_431f_9811_f97e41935c40.slice/crio-c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4 WatchSource:0}: Error finding container c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4: Status 404 returned error can't find the container with id c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4 Nov 02 06:16:14.090232 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.090147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:14.090232 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.090189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:14.090421 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.090299 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:14.090421 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.090366 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:16.090347901 +0000 UTC m=+5.338912431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:14.090421 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.090299 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:14.090552 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.090435 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:16.090424143 +0000 UTC m=+5.338988686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:14.191256 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.191217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:14.191846 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.191404 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:14.191846 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.191432 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:14.191846 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.191446 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:14.191846 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.191505 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:16.191486502 +0000 UTC m=+5.440051046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:14.311042 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.311011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:14.311204 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.311142 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:14.311524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.311502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:14.311630 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.311582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:14.311669 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.311632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:14.311704 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:14.311682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:14.318577 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.318539 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kcl2z" event={"ID":"37012b0f-1510-431f-9811-f97e41935c40","Type":"ContainerStarted","Data":"c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4"} Nov 02 06:16:14.319887 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.319864 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29z9j" event={"ID":"fe8c4326-8500-4a72-848f-7281b21b4853","Type":"ContainerStarted","Data":"e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb"} Nov 02 06:16:14.321130 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.321108 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7lsf6" event={"ID":"0eb289fe-8e1a-4abe-a37b-4cc6ac343150","Type":"ContainerStarted","Data":"c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93"} Nov 02 06:16:14.322682 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.322659 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9gfs" event={"ID":"2abb23a4-08db-4159-8715-df445adc7a71","Type":"ContainerStarted","Data":"e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4"} Nov 02 06:16:14.324075 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.324050 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gqnpb" event={"ID":"853c435f-0786-450f-bdd9-f95cc551ba78","Type":"ContainerStarted","Data":"05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c"} Nov 02 06:16:14.325271 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.325251 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerStarted","Data":"73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902"} Nov 02 06:16:14.327040 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.327020 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" event={"ID":"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4","Type":"ContainerStarted","Data":"dddb6a1eea2cb25027d80dd2683e2fdf8565dd5a1fba441612510e13070fd835"} Nov 02 06:16:14.328286 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.328262 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b"} Nov 02 06:16:14.331066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.331047 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" event={"ID":"ff9ab1b6-621a-430c-9669-1071af47a675","Type":"ContainerStarted","Data":"87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f"} Nov 02 06:16:14.442661 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.442584 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:11:12 +0000 UTC" deadline="2028-07-16 17:07:28.683675458 +0000 UTC" Nov 02 06:16:14.442661 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:14.442623 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14938h51m14.241054391s" Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.108189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.108276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.108459 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.108538 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.108524118 +0000 UTC m=+9.357088659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.108859 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:16.108971 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.108896 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.108885794 +0000 UTC m=+9.357450337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:16.209761 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.209479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:16.209761 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.209664 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:16.209761 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.209680 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:16.209761 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.209689 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:16.209761 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.209730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.209718057 +0000 UTC m=+9.458282599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:16.310466 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.310365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:16.310681 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.310495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:16.310681 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.310537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:16.310681 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.310658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:16.310867 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:16.310712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:16.310867 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:16.310789 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:17.355816 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:17.355539 2579 generic.go:356] "Generic (PLEG): container finished" podID="bd6c06b094cb0b06c560a952f3f0bc9d" containerID="6a248642ba3fd57a4f6beb75efd8114cd59d8437e84dc0f34f011acd10d07cab" exitCode=0 Nov 02 06:16:17.355816 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:17.355821 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerDied","Data":"6a248642ba3fd57a4f6beb75efd8114cd59d8437e84dc0f34f011acd10d07cab"} Nov 02 06:16:17.375959 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:17.375929 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerStarted","Data":"2229b3416aafeeeebedd2e6b315b31746e76f82efcf16eca6503f77eceb7b630"} Nov 02 06:16:17.385129 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:17.384766 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"05718cfa86289f66de168b803f5f9362","Type":"ContainerStarted","Data":"af6327a04ee741b95aa057554735df7633bf3d9c7ce9ff15e991ea1636fd8396"} Nov 02 06:16:17.567936 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:17.566970 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" podStartSLOduration=6.5669528360000005 podStartE2EDuration="6.566952836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:17.566817776 +0000 UTC m=+6.815382327" watchObservedRunningTime="2026-11-02 06:16:17.566952836 +0000 UTC m=+6.815517388" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.311291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:18.311417 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.311568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:18.311654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.311723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:18.312019 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:18.311815 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:18.397910 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.397875 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="2229b3416aafeeeebedd2e6b315b31746e76f82efcf16eca6503f77eceb7b630" exitCode=0 Nov 02 06:16:18.398310 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.397971 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"2229b3416aafeeeebedd2e6b315b31746e76f82efcf16eca6503f77eceb7b630"} Nov 02 06:16:18.406648 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.406621 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerStarted","Data":"a96fb4de76c154622e10bfc1daedfef5c57b1eb43c539c541d6d7d248fc19a6c"} Nov 02 06:16:18.635892 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:18.635790 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" podStartSLOduration=7.635771597 podStartE2EDuration="7.635771597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:18.635278155 +0000 UTC m=+7.883842707" watchObservedRunningTime="2026-11-02 06:16:18.635771597 +0000 UTC m=+7.884336150" Nov 02 06:16:19.414417 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:19.414381 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gqnpb" event={"ID":"853c435f-0786-450f-bdd9-f95cc551ba78","Type":"ContainerStarted","Data":"b044a92de9536d9476382af02ffec9d0e73571b9c781aac01d836f524df337e5"} Nov 02 06:16:20.149237 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.149207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:20.149338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.149250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:20.149424 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.149409 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:20.149478 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.149471 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.149453005 +0000 UTC m=+17.398017549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:20.149549 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.149536 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:20.149600 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.149575 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.149563938 +0000 UTC m=+17.398128481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:20.209403 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.209378 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:20.209877 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.209856 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:20.249562 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.249538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:20.249701 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.249677 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:20.249701 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.249696 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:20.249811 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.249705 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:20.249811 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.249745 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.249733501 +0000 UTC m=+17.498298030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:20.310939 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.310900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:20.311073 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.311017 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:20.311134 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.311099 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:20.311191 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.311158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:20.311241 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.311215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:20.311288 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:20.311269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:20.314792 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.314734 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gqnpb" podStartSLOduration=4.689229627 podStartE2EDuration="9.314716858s" totalImagesPullingTime="4.625487231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:19.520969312 +0000 UTC m=+8.769533869" watchObservedRunningTime="2026-11-02 06:16:20.314716858 +0000 UTC m=+9.563281414" Nov 02 06:16:20.418626 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.418089 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9gfs" event={"ID":"2abb23a4-08db-4159-8715-df445adc7a71","Type":"ContainerStarted","Data":"01c1ab4c0f17d327f3197f88242e39d9e80ad219576656d22e57080024a66dc4"} Nov 02 06:16:20.421545 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:20.421464 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" event={"ID":"ff9ab1b6-621a-430c-9669-1071af47a675","Type":"ContainerStarted","Data":"fa7fa7e15f2435acf88b6d45475989a3e2c01c1db5a6ab49a955b02bfc09d07a"} Nov 02 06:16:21.424519 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:21.424489 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:16:22.236453 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.236431 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:16:22.242573 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.242486 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:16:22.236448886Z","UUID":"efdc5ecd-a30f-49d7-81dd-ec116a164002","Handler":null,"Name":"","Endpoint":""} Nov 02 06:16:22.244544 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.244125 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:16:22.244544 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.244146 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:16:22.310934 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.310905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:22.311100 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.310974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:22.311100 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.311011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:22.311220 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:22.311113 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:22.311220 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:22.311182 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:22.311316 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:22.311251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:22.428005 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.427916 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" event={"ID":"ff9ab1b6-621a-430c-9669-1071af47a675","Type":"ContainerStarted","Data":"5fcba3bda3e9c88bd841098a02772856c355af1f45667442191f7fb0b92b6646"} Nov 02 06:16:22.430112 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.430065 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kcl2z" event={"ID":"37012b0f-1510-431f-9811-f97e41935c40","Type":"ContainerStarted","Data":"a11566ef87706df159e114dd8f2dbfbbfac910b95c4dc37ce6361f437bbb8b11"} Nov 02 06:16:22.432073 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.431848 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29z9j" event={"ID":"fe8c4326-8500-4a72-848f-7281b21b4853","Type":"ContainerStarted","Data":"5e6487c98a9925a5c90d133e921bf0632bfce5fc3a585b0663e53671d6986e2e"} Nov 02 06:16:22.538651 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.538603 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v9gfs" podStartSLOduration=5.914435697 podStartE2EDuration="11.538590337s" totalImagesPullingTime="5.62415464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:20.524730685 +0000 UTC m=+9.773295237" watchObservedRunningTime="2026-11-02 06:16:22.538590337 +0000 UTC m=+11.787154946" Nov 02 06:16:22.646045 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.645743 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kcl2z" podStartSLOduration=3.266423843 podStartE2EDuration="11.645725114s" totalImagesPullingTime="8.379301271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:22.537939389 +0000 UTC m=+11.786503941" watchObservedRunningTime="2026-11-02 06:16:22.645725114 +0000 UTC m=+11.894289665" Nov 02 06:16:22.646455 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:22.646403 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-29z9j" podStartSLOduration=3.685291091 podStartE2EDuration="11.646393388s" totalImagesPullingTime="7.961102297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:22.645203607 +0000 UTC m=+11.893768161" watchObservedRunningTime="2026-11-02 06:16:22.646393388 +0000 UTC m=+11.894957940" Nov 02 06:16:23.325207 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:23.325178 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kcl2z_37012b0f-1510-431f-9811-f97e41935c40/dns-node-resolver/0.log" Nov 02 06:16:23.437890 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:23.437850 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7lsf6" event={"ID":"0eb289fe-8e1a-4abe-a37b-4cc6ac343150","Type":"ContainerStarted","Data":"54104d5ab0750387cdb80cca75f766ddf6efcb295fb735f18b749fdb40429601"} Nov 02 06:16:24.164967 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.164906 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9gfs_2abb23a4-08db-4159-8715-df445adc7a71/node-ca/0.log" Nov 02 06:16:24.310724 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.310695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:24.310881 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.310767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:24.310999 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.310881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:24.310999 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:24.310898 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:24.311094 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:24.310979 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:24.311132 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:24.311097 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:24.441544 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.441459 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="911f319e5473976af60f41f128bffd7990a3caac8b6f81154b641432b7fb3bc8" exitCode=0 Nov 02 06:16:24.442223 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.441550 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"911f319e5473976af60f41f128bffd7990a3caac8b6f81154b641432b7fb3bc8"} Nov 02 06:16:24.443605 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.443436 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" event={"ID":"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4","Type":"ContainerStarted","Data":"a0e4df0493d6d0bce65744dfebcb0f016cd3162ee7ec2293bb8c7718bf02cab7"} Nov 02 06:16:24.445937 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.445914 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" event={"ID":"ff9ab1b6-621a-430c-9669-1071af47a675","Type":"ContainerStarted","Data":"220fb06ce87c9f79208562ff81a04cec0bf7b05d43aa6f292cb4abdac0e152e6"} Nov 02 06:16:24.585870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.585827 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7lsf6" podStartSLOduration=5.519399447 podStartE2EDuration="13.585815003s" totalImagesPullingTime="8.066415556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:23.549506924 +0000 UTC m=+12.798071476" watchObservedRunningTime="2026-11-02 06:16:24.585815003 +0000 UTC m=+13.834379554" Nov 02 06:16:24.695286 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.695199 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" podStartSLOduration=3.446907969 podStartE2EDuration="13.695185188s" totalImagesPullingTime="10.248277219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:24.694850707 +0000 UTC m=+13.943415258" watchObservedRunningTime="2026-11-02 06:16:24.695185188 +0000 UTC m=+13.943749739" Nov 02 06:16:24.802775 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:24.802727 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" podStartSLOduration=3.803955508 podStartE2EDuration="13.802709625s" totalImagesPullingTime="9.998754117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:24.80225412 +0000 UTC m=+14.050818672" watchObservedRunningTime="2026-11-02 06:16:24.802709625 +0000 UTC m=+14.051274178" Nov 02 06:16:26.311089 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:26.310868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:26.311613 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:26.310885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:26.311613 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:26.311190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:26.311613 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:26.311268 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:26.311613 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:26.310895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:26.311825 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:26.311805 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:26.453364 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:26.453284 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="6a00855691fd0f45884c4425b013e425f93b29504aa624d7ae31a46e573f1b1b" exitCode=0 Nov 02 06:16:26.453364 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:26.453340 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"6a00855691fd0f45884c4425b013e425f93b29504aa624d7ae31a46e573f1b1b"} Nov 02 06:16:28.216675 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.216646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:28.217116 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.216687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:28.217116 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.216821 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:28.217116 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.216836 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:28.217116 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.216898 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:44.216877538 +0000 UTC m=+33.465442078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:28.217116 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.216919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:44.216911455 +0000 UTC m=+33.465475985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:28.311240 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.311211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:28.311361 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.311267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:28.311361 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.311266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:28.311454 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.311377 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:28.311454 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.311434 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:28.311550 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.311511 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:28.317338 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:28.317312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:28.317556 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.317453 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:28.317556 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.317477 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:28.317556 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.317487 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:28.317556 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:28.317536 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:44.317519345 +0000 UTC m=+33.566083874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:29.461613 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:29.461584 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="566ef63d9f18245b8b5ac28fafc091aa718772ecf5c875b01db44127b99654c6" exitCode=0 Nov 02 06:16:29.462007 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:29.461627 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"566ef63d9f18245b8b5ac28fafc091aa718772ecf5c875b01db44127b99654c6"} Nov 02 06:16:30.310911 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.310736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:30.311064 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.310809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:30.311064 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:30.311005 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:30.311138 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.310847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:30.311138 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:30.311104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:30.311232 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:30.311140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:30.466511 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466434 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"68450c31fff5299b3c93d410bfc84cc1bcb402f2f437b3cbc053002129b58fa4"} Nov 02 06:16:30.466895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466510 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"51cadda300fc921280987e9d25d807200785a14132740db18fe4f9e2aebeb055"} Nov 02 06:16:30.466895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466554 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"ef807469ddf8a83984eced02d958f39b5b5779f315e0fb0b06c157a34a80933d"} Nov 02 06:16:30.466895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466600 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"d00ec3aeb938a1c2dc14e3f6b29d60053eb252318ec47b28b6fab5b120c9790a"} Nov 02 06:16:30.466895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466650 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"9be679d61d4a684b7b8c82dfa9713d43d017be4490c9581a050000ab917f7ff6"} Nov 02 06:16:30.466895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:30.466695 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"4ccc80ec93eb42659c796b040fae7800f25f1deec11edf4989bba63c16f9a5f5"} Nov 02 06:16:32.310365 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:32.310341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:32.310710 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:32.310341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:32.310710 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:32.310460 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:32.310710 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:32.310344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:32.310710 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:32.310578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:32.310710 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:32.310626 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:32.474290 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:32.474259 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"cd4ee11123e28831a4bf3bebf36a636f7a29246ad7c8eddfa1d232655e347c45"} Nov 02 06:16:33.478328 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:33.478297 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="a6d7a54875d56724e3917f5cc353e9bfe95f5be57dbf7450568cc84d27155c00" exitCode=0 Nov 02 06:16:33.478819 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:33.478361 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"a6d7a54875d56724e3917f5cc353e9bfe95f5be57dbf7450568cc84d27155c00"} Nov 02 06:16:34.310362 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.310333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:34.310362 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.310359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:34.310605 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.310332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:34.310605 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:34.310445 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:34.310605 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:34.310497 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:34.310605 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:34.310598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:34.484251 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.484218 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="c56d319523a90db1f3a399260ef6751e53b7fbcc70747373b876292e3feb97e3" exitCode=0 Nov 02 06:16:34.484849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.484296 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerDied","Data":"c56d319523a90db1f3a399260ef6751e53b7fbcc70747373b876292e3feb97e3"} Nov 02 06:16:34.488671 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.488647 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" event={"ID":"daa211aa-bb54-4375-bb65-793ae2e67721","Type":"ContainerStarted","Data":"413130bafecc1fbabd49ed6d477511f55e30b6112670b70698d8f35a0f15d778"} Nov 02 06:16:34.488972 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.488954 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:34.489077 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.489030 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:34.506153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.506134 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:34.741384 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:34.741339 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" podStartSLOduration=7.527248233 podStartE2EDuration="23.741326288s" totalImagesPullingTime="16.214078055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:34.740891129 +0000 UTC m=+23.989455680" watchObservedRunningTime="2026-11-02 06:16:34.741326288 +0000 UTC m=+23.989890877" Nov 02 06:16:35.494256 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.494215 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88wtj" event={"ID":"fd25d9ad-68e0-49c8-a9cf-0ca18a976680","Type":"ContainerStarted","Data":"12b530c07537e5a154f41ec1937120e11e2839e5e424c2a3cea43b92b1ad8b7f"} Nov 02 06:16:35.494718 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.494691 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:35.508917 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.508895 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:16:35.587569 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.587547 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:35.587669 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.587656 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:16:35.588194 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.588170 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:16:35.605443 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:35.605406 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-88wtj" podStartSLOduration=8.740639659 podStartE2EDuration="24.60539631s" totalImagesPullingTime="15.864756651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:35.604873423 +0000 UTC m=+24.853437973" watchObservedRunningTime="2026-11-02 06:16:35.60539631 +0000 UTC m=+24.853960860" Nov 02 06:16:36.310241 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:36.310211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:36.310396 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:36.310329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:36.310396 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:36.310339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:36.310396 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:36.310331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:36.310481 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:36.310402 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:36.310515 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:36.310497 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:38.310610 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:38.310409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:38.311025 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:38.310443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:38.311025 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:38.310679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:38.311025 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:38.310489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:38.311025 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:38.310749 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:38.311025 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:38.310814 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:40.310560 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:40.310532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:40.310560 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:40.310553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:40.311123 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:40.310532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:40.311123 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:40.310654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:40.311123 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:40.310703 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:40.311123 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:40.310741 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:42.310640 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:42.310607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:42.310640 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:42.310629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:42.311096 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:42.310709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:42.311096 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:42.310722 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:42.311096 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:42.310762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:42.311096 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:42.310834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:43.100200 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.100024 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vd4bl"] Nov 02 06:16:43.100200 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.100169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:43.100388 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:43.100264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:43.101186 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.101158 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h5h5g"] Nov 02 06:16:43.101354 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.101217 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s4zwp"] Nov 02 06:16:43.101354 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.101285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:43.101521 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:43.101383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:43.101521 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:43.101410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:43.101521 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:43.101480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:44.232195 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:44.232165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:44.232504 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:44.232202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:44.232504 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.232302 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:44.232504 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.232350 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret podName:fab2c88a-127f-4391-870a-f9757c5f9b04 nodeName:}" failed. No retries permitted until 2026-11-02 06:17:16.232333725 +0000 UTC m=+65.480898254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret") pod "global-pull-secret-syncer-h5h5g" (UID: "fab2c88a-127f-4391-870a-f9757c5f9b04") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:44.232504 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.232302 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:44.232504 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.232420 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs podName:e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69 nodeName:}" failed. No retries permitted until 2026-11-02 06:17:16.232405736 +0000 UTC m=+65.480970265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs") pod "network-metrics-daemon-s4zwp" (UID: "e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:44.310568 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:44.310541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:44.310699 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.310673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:44.310759 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:44.310715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:44.310798 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.310782 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:44.332713 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:44.332693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:44.332831 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.332819 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:44.332868 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.332835 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:44.332868 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.332843 2579 projected.go:196] Error preparing data for projected volume kube-api-access-brvn6 for pod openshift-network-diagnostics/network-check-target-vd4bl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:44.332931 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:44.332884 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6 podName:64c2f13d-6a20-4247-b926-ad21d58dc644 nodeName:}" failed. No retries permitted until 2026-11-02 06:17:16.332868056 +0000 UTC m=+65.581432601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-brvn6" (UniqueName: "kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6") pod "network-check-target-vd4bl" (UID: "64c2f13d-6a20-4247-b926-ad21d58dc644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:45.311306 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:45.311283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:45.311726 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:45.311415 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:46.310999 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:46.310965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:46.311153 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:46.311022 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:46.311153 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:46.311096 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd4bl" podUID="64c2f13d-6a20-4247-b926-ad21d58dc644" Nov 02 06:16:46.311220 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:46.311205 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5h5g" podUID="fab2c88a-127f-4391-870a-f9757c5f9b04" Nov 02 06:16:47.311230 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.311204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:47.311527 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:16:47.311311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4zwp" podUID="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" Nov 02 06:16:47.541760 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.541739 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeReady" Nov 02 06:16:47.541899 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.541839 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 06:16:47.689940 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.689915 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hmlst"] Nov 02 06:16:47.715957 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.715933 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqq7v"] Nov 02 06:16:47.716114 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.716090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.732144 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.731981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:47.739179 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.739159 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-htss2"] Nov 02 06:16:47.754980 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.754962 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqq7v"] Nov 02 06:16:47.755074 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.755043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.756884 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.756862 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmlst"] Nov 02 06:16:47.765928 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.765909 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:16:47.766029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.765960 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:16:47.766133 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.766119 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:16:47.766167 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.766145 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:16:47.766167 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.766157 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:16:47.785093 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.785076 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:16:47.785173 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.785093 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:16:47.785173 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.785149 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:16:47.799897 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.799842 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:16:47.800884 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.800862 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htss2"] Nov 02 06:16:47.861133 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.861224 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkxk\" (UniqueName: \"kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.861224 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.861224 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.861224 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.861365 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:47.861397 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.861397 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.861457 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.861457 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pkr\" (UniqueName: \"kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:47.861545 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.861531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fx8\" (UniqueName: \"kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.866609 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.866592 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:16:47.866828 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.866815 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:16:47.866919 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.866908 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:16:47.962196 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fx8\" (UniqueName: \"kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962315 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.962315 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkxk\" (UniqueName: \"kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.962315 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.962468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:47.962468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.962468 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962700 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962700 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pkr\" (UniqueName: \"kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:47.962700 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962545 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.962917 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.963020 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.962977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.963227 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.963201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.963305 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.963248 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.966188 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.966168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:47.966293 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.966252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:47.966346 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:47.966292 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:48.015918 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.015893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fx8\" (UniqueName: \"kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:48.016022 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.015925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkxk\" (UniqueName: \"kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:16:48.016410 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.016394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pkr\" (UniqueName: \"kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:48.061410 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.061367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:16:48.083287 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.083268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:16:48.118870 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.118848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htss2" Nov 02 06:16:48.310670 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.310644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:16:48.310789 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.310676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:16:48.359895 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.359870 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:48.362509 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.362491 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:16:48.362930 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.362914 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:16:48.363008 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.362997 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:16:48.462601 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.462574 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:48.554800 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.554742 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqq7v"] Nov 02 06:16:48.557905 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.557887 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htss2"] Nov 02 06:16:48.558983 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:48.558967 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmlst"] Nov 02 06:16:48.560425 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:48.560403 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c11852f_4b27_4820_bfae_f65e13a3ceaa.slice/crio-b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99 WatchSource:0}: Error finding container b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99: Status 404 returned error can't find the container with id b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99 Nov 02 06:16:48.561434 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:48.561415 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe97868d_2a06_4ec7_a874_d88a2dfd0978.slice/crio-9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589 WatchSource:0}: Error finding container 9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589: Status 404 returned error can't find the container with id 9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589 Nov 02 06:16:48.596796 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:48.596773 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58618324_800f_4705_87d8_d68e27813002.slice/crio-e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e WatchSource:0}: Error finding container e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e: Status 404 returned error can't find the container with id e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e Nov 02 06:16:49.310768 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.310733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:16:49.360438 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.360407 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:49.360438 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.360407 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:49.522246 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.522214 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htss2" event={"ID":"58618324-800f-4705-87d8-d68e27813002","Type":"ContainerStarted","Data":"e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e"} Nov 02 06:16:49.523393 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.523366 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqq7v" event={"ID":"2c11852f-4b27-4820-bfae-f65e13a3ceaa","Type":"ContainerStarted","Data":"b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99"} Nov 02 06:16:49.524657 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.524637 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmlst" event={"ID":"fe97868d-2a06-4ec7-a874-d88a2dfd0978","Type":"ContainerStarted","Data":"e562c0afdc406afa72f0811e57586f4f8e4815ef8ae619f4accb943b2132ad42"} Nov 02 06:16:49.524762 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:49.524679 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmlst" event={"ID":"fe97868d-2a06-4ec7-a874-d88a2dfd0978","Type":"ContainerStarted","Data":"9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589"} Nov 02 06:16:50.253820 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.253790 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc"] Nov 02 06:16:50.260751 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.260721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:50.310350 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.310326 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc"] Nov 02 06:16:50.355479 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.355456 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2dzzq\"" Nov 02 06:16:50.355746 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.355728 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 06:16:50.381169 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.381146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7bxsc\" (UID: \"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:50.481972 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.481940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7bxsc\" (UID: \"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:50.484685 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.484663 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7bxsc\" (UID: \"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:50.528790 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.528709 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmlst" event={"ID":"fe97868d-2a06-4ec7-a874-d88a2dfd0978","Type":"ContainerStarted","Data":"5625cbad2c0d4670ffa666be4fefd992b9cc3630a07978d9eabd27981100879d"} Nov 02 06:16:50.577976 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.577952 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:50.963212 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:50.963184 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc"] Nov 02 06:16:50.967239 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:50.967212 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bc39c2_67ef_463a_ac5c_cc48dc9e00e8.slice/crio-31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44 WatchSource:0}: Error finding container 31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44: Status 404 returned error can't find the container with id 31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44 Nov 02 06:16:51.533403 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:51.533360 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" event={"ID":"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8","Type":"ContainerStarted","Data":"31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44"} Nov 02 06:16:52.541223 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:52.541182 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htss2" event={"ID":"58618324-800f-4705-87d8-d68e27813002","Type":"ContainerStarted","Data":"d08688c0e275031605e92c48f59250f96ad77552534c40cbc156e3f03e26f41c"} Nov 02 06:16:53.545934 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.545776 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmlst" event={"ID":"fe97868d-2a06-4ec7-a874-d88a2dfd0978","Type":"ContainerStarted","Data":"f3b8305ceb54469a98302653fa58041a024b2f0ecb7099037ecaa06d0140bf21"} Nov 02 06:16:53.547359 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.547336 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htss2" event={"ID":"58618324-800f-4705-87d8-d68e27813002","Type":"ContainerStarted","Data":"02dc82fa33d8f13a4cb08bf025b92a9f29d6e13def985a9b4f0493db591f86ea"} Nov 02 06:16:53.547475 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.547464 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-htss2" Nov 02 06:16:53.694226 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.694168 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-htss2" podStartSLOduration=2.973105856 podStartE2EDuration="6.694156042s" totalImagesPullingTime="3.721050186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:53.693655507 +0000 UTC m=+42.942220058" watchObservedRunningTime="2026-11-02 06:16:53.694156042 +0000 UTC m=+42.942720593" Nov 02 06:16:53.899047 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.898950 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kkplp"] Nov 02 06:16:53.913922 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.913897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:53.961182 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.961035 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kkplp"] Nov 02 06:16:53.963637 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.963621 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vdcff\"" Nov 02 06:16:53.964378 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.964353 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.964378 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:53.964368 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 06:16:54.008972 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.008948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzs2k\" (UniqueName: \"kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.009094 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.009079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.110089 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.110059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.110247 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.110109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzs2k\" (UniqueName: \"kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.110551 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.110529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.165326 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.165293 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzs2k\" (UniqueName: \"kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.245061 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.245030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:16:54.552777 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.552683 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqq7v" event={"ID":"2c11852f-4b27-4820-bfae-f65e13a3ceaa","Type":"ContainerStarted","Data":"6e7a83c6c1cf493bb74757bb6554c570157838eadd6a527d9bdd333710e98701"} Nov 02 06:16:54.554268 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.554230 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" event={"ID":"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8","Type":"ContainerStarted","Data":"5e4e0c2a8e87bf9289abde897a13f38df54ae2faa99d3ce05718a584d760d12c"} Nov 02 06:16:54.554403 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.554347 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:54.558966 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.558946 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:16:54.658817 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.658771 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqq7v" podStartSLOduration=2.250369883 podStartE2EDuration="7.658759715s" totalImagesPullingTime="5.408389832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:54.65825706 +0000 UTC m=+43.906821612" watchObservedRunningTime="2026-11-02 06:16:54.658759715 +0000 UTC m=+43.907324266" Nov 02 06:16:54.717255 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.717232 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kkplp"] Nov 02 06:16:54.756355 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:54.756328 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b9b535_bd12_4e37_8b22_bf77f7cd93db.slice/crio-fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e WatchSource:0}: Error finding container fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e: Status 404 returned error can't find the container with id fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e Nov 02 06:16:54.768845 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.768805 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hmlst" podStartSLOduration=3.078172408 podStartE2EDuration="7.768792941s" totalImagesPullingTime="4.690620533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:54.767933803 +0000 UTC m=+44.016498365" watchObservedRunningTime="2026-11-02 06:16:54.768792941 +0000 UTC m=+44.017357489" Nov 02 06:16:54.875129 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:54.875060 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" podStartSLOduration=1.8850983860000001 podStartE2EDuration="4.875048893s" totalImagesPullingTime="2.989950507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:54.874265491 +0000 UTC m=+44.122830064" watchObservedRunningTime="2026-11-02 06:16:54.875048893 +0000 UTC m=+44.123613443" Nov 02 06:16:55.558947 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.558908 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kkplp" event={"ID":"80b9b535-bd12-4e37-8b22-bf77f7cd93db","Type":"ContainerStarted","Data":"fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e"} Nov 02 06:16:55.852228 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.852151 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-shhdr"] Nov 02 06:16:55.854275 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.854254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:55.903660 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.903640 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:16:55.933027 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.932998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:55.933134 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.933080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:55.933134 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.933125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:55.933218 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.933208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblwh\" (UniqueName: \"kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:55.950141 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.950124 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 06:16:55.950247 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.950202 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 06:16:55.950247 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.950228 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:16:55.950351 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.950302 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 06:16:55.950351 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.950342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l2vqf\"" Nov 02 06:16:55.997370 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:55.997346 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-shhdr"] Nov 02 06:16:56.034280 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.034257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblwh\" (UniqueName: \"kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.034363 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.034310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.034404 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.034359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.034434 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.034399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.035154 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.035135 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.037421 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.037398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.037421 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.037408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.088863 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.088835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblwh\" (UniqueName: \"kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.166232 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.166205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:16:56.527686 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:56.527602 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-shhdr"] Nov 02 06:16:56.562198 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:16:56.562164 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5c9a49_a662_4b53_bbe8_31fc85b64434.slice/crio-4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d WatchSource:0}: Error finding container 4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d: Status 404 returned error can't find the container with id 4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d Nov 02 06:16:57.567787 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:57.567751 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" event={"ID":"ba5c9a49-a662-4b53-bbe8-31fc85b64434","Type":"ContainerStarted","Data":"4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d"} Nov 02 06:16:59.576492 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:59.576455 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" event={"ID":"ba5c9a49-a662-4b53-bbe8-31fc85b64434","Type":"ContainerStarted","Data":"f1a25a81ab60b2e36097508fd750187ad5455a983e0b47d4dea550d8961e8de5"} Nov 02 06:16:59.577012 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:59.576523 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" event={"ID":"ba5c9a49-a662-4b53-bbe8-31fc85b64434","Type":"ContainerStarted","Data":"45ebb22900dac95a84c28b99a6703a415a2ff2bd7f7c5f0e825960c8c434b5d9"} Nov 02 06:16:59.684199 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:16:59.684144 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" podStartSLOduration=2.256421572 podStartE2EDuration="4.684131637s" totalImagesPullingTime="2.427710065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:59.683602171 +0000 UTC m=+48.932166746" watchObservedRunningTime="2026-11-02 06:16:59.684131637 +0000 UTC m=+48.932696188" Nov 02 06:17:01.995782 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:01.995753 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:17:01.997921 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:01.997904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.047233 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.047172 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 06:17:02.078555 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078668 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078725 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078725 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078798 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078838 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.078883 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.078865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.093469 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.093449 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 06:17:02.094029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.093703 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 06:17:02.094029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.093790 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 06:17:02.094029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.093963 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 06:17:02.094029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.094024 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5mfb9\"" Nov 02 06:17:02.095195 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.095170 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:17:02.180054 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180166 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180166 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180166 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180166 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180360 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180360 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180590 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.180971 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.180952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.181066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.181018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.181066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.181056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.182803 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.182773 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.183357 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.183328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.233674 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.233647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.309074 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.309017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:02.524626 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.524595 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz"] Nov 02 06:17:02.527046 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.527025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.540710 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.540690 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7xg9n"] Nov 02 06:17:02.542804 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.542789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.577456 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.577407 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 06:17:02.584195 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584277 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584277 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584373 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6p26\" (UniqueName: \"kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584373 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584373 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584373 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584572 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584572 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.584572 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584572 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fzk\" (UniqueName: \"kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.584690 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.584586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.587838 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.587816 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz"] Nov 02 06:17:02.650035 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.650011 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 06:17:02.666279 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666257 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 06:17:02.666397 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666265 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 06:17:02.666397 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666291 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9sdpq\"" Nov 02 06:17:02.666397 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666288 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 06:17:02.666500 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666474 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 06:17:02.666756 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.666743 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbcc6\"" Nov 02 06:17:02.685274 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685376 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685376 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685376 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fzk\" (UniqueName: \"kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.685770 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.685746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p26\" (UniqueName: \"kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.687250 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.687232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.687699 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.687303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.687699 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.687671 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.687928 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.687902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.688074 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.688021 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:17:02.688135 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.688085 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.688194 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.688134 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.689113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.688356 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.689113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.688725 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.689113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.689011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.689113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.689077 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.690446 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.690425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.690512 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.690490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.692013 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.691968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.693637 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:02.693612 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2cc1bb_2724_4b77_80a6_20be4eb4ca52.slice/crio-8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7 WatchSource:0}: Error finding container 8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7: Status 404 returned error can't find the container with id 8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7 Nov 02 06:17:02.739702 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.739678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p26\" (UniqueName: \"kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.744780 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.744755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fzk\" (UniqueName: \"kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.840209 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.840157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:02.856103 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:02.856080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:02.867893 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:02.867860 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ebe6ec_1b67_42bd_8f17_873ea5c799a4.slice/crio-5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d WatchSource:0}: Error finding container 5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d: Status 404 returned error can't find the container with id 5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d Nov 02 06:17:03.208746 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:03.208716 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz"] Nov 02 06:17:03.216281 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:03.216256 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb424524e_6361_4ccb_9de6_58c78d4021fa.slice/crio-85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c WatchSource:0}: Error finding container 85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c: Status 404 returned error can't find the container with id 85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c Nov 02 06:17:03.557567 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:03.557486 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-htss2" Nov 02 06:17:03.595023 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:03.594995 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" event={"ID":"b424524e-6361-4ccb-9de6-58c78d4021fa","Type":"ContainerStarted","Data":"85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c"} Nov 02 06:17:03.596029 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:03.596008 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xg9n" event={"ID":"20ebe6ec-1b67-42bd-8f17-873ea5c799a4","Type":"ContainerStarted","Data":"5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d"} Nov 02 06:17:03.597022 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:03.597000 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f964984-9rxlg" event={"ID":"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52","Type":"ContainerStarted","Data":"8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7"} Nov 02 06:17:04.085542 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.085486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqnpb_853c435f-0786-450f-bdd9-f95cc551ba78/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:04.331074 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.331027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqnpb_853c435f-0786-450f-bdd9-f95cc551ba78/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:04.752862 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.752826 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:04.756247 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.756228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.761269 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.761232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqnpb_853c435f-0786-450f-bdd9-f95cc551ba78/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:04.806464 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.806431 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:04.806915 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.806880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807011 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.806922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807011 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.806964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807124 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807124 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807124 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807124 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807257 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807257 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807257 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807367 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807367 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.807367 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.807309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.852851 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.852820 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tx8jf\"" Nov 02 06:17:04.853048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.852905 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 06:17:04.853048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.852963 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 06:17:04.853141 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853059 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 06:17:04.853141 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853115 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 06:17:04.853205 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853184 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 06:17:04.853342 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853327 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 06:17:04.853491 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853474 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 06:17:04.853592 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.853564 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 06:17:04.908066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908212 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908212 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908212 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908212 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908699 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.908699 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.908461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.910290 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.910255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.910940 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.910647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.914445 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.914422 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.915417 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.915379 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.916644 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.916618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.916758 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.916716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.916977 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.916951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.917284 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.917262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.917354 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.917289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.918610 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.918541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.919739 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.919712 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.945288 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.945262 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 06:17:04.951749 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.951687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:04.963038 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:04.962978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:05.078655 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.078582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:05.115949 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.115922 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml"] Nov 02 06:17:05.118920 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.118895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.183977 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.183945 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml"] Nov 02 06:17:05.211595 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211754 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgkr\" (UniqueName: \"kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211754 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211754 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211754 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211943 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.211943 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.212066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.211958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.212066 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.212048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.257577 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8hjs78d5780tl\"" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.257665 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.257732 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.257810 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.257899 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 06:17:05.258113 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.258080 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 06:17:05.258660 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.258143 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fq2h9\"" Nov 02 06:17:05.313318 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313484 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313484 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313484 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313484 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313484 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313731 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313731 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.313731 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.313650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgkr\" (UniqueName: \"kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.314841 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.314813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.315789 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.315760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.316721 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.316674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.323316 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.323254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.324584 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.324524 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.325470 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.325426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.326711 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.326679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.326969 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.326947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.368619 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.368530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgkr\" (UniqueName: \"kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.446271 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.446072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:05.526316 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.526215 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:05.549463 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:05.549388 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1e42d4_ad24_4090_bf7a_1af618a6ba1c.slice/crio-a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09 WatchSource:0}: Error finding container a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09: Status 404 returned error can't find the container with id a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09 Nov 02 06:17:05.618038 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.618006 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09"} Nov 02 06:17:05.620872 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.620814 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" event={"ID":"b424524e-6361-4ccb-9de6-58c78d4021fa","Type":"ContainerStarted","Data":"fbf8931f2527cb3725eb7beb1b0d99a6385a1f99cb55b3061aa4d63ddc9c7d70"} Nov 02 06:17:05.852736 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:05.852705 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml"] Nov 02 06:17:05.860964 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:05.860934 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206c88d3_cdff_496a_ac41_e8e7feca0ac8.slice/crio-ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d WatchSource:0}: Error finding container ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d: Status 404 returned error can't find the container with id ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d Nov 02 06:17:06.629015 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.628963 2579 generic.go:356] "Generic (PLEG): container finished" podID="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" containerID="4bc0c5add56952e7cd1836c55c802ca5648a296f56fa093bb908299d3aeddb48" exitCode=0 Nov 02 06:17:06.629486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.629085 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xg9n" event={"ID":"20ebe6ec-1b67-42bd-8f17-873ea5c799a4","Type":"ContainerDied","Data":"4bc0c5add56952e7cd1836c55c802ca5648a296f56fa093bb908299d3aeddb48"} Nov 02 06:17:06.631215 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.631185 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d"} Nov 02 06:17:06.634442 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.634416 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" event={"ID":"b424524e-6361-4ccb-9de6-58c78d4021fa","Type":"ContainerStarted","Data":"2c5ed74e866e8d843db16639b67edb0e6d7be9a828063101799c7fd57234c65b"} Nov 02 06:17:06.634542 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.634472 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" event={"ID":"b424524e-6361-4ccb-9de6-58c78d4021fa","Type":"ContainerStarted","Data":"e1a2050ceb224d86b7cc93fab51da237fa938f5d17a054c9ca42f4aeea05792c"} Nov 02 06:17:06.859622 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:06.859566 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" podStartSLOduration=2.600054381 podStartE2EDuration="4.859551152s" totalImagesPullingTime="2.259496771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:06.858385968 +0000 UTC m=+56.106950509" watchObservedRunningTime="2026-11-02 06:17:06.859551152 +0000 UTC m=+56.108115705" Nov 02 06:17:07.132431 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.132382 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-skv9z"] Nov 02 06:17:07.135760 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.135735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.185959 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.185923 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 06:17:07.186120 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.186057 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8h274\"" Nov 02 06:17:07.190904 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.190843 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-skv9z"] Nov 02 06:17:07.233365 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.233071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.233365 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.233155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqbc\" (UniqueName: \"kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.329835 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.328488 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64f9df6957-68857"] Nov 02 06:17:07.332150 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.332097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.334429 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.333630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.334429 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.333682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqbc\" (UniqueName: \"kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.350264 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.350237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.382660 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.382570 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 06:17:07.382660 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.382652 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 06:17:07.382860 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.382828 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-144i712719v8e\"" Nov 02 06:17:07.382927 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.382908 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 06:17:07.383062 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.382937 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qnwd4\"" Nov 02 06:17:07.387866 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.387802 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64f9df6957-68857"] Nov 02 06:17:07.393612 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.393586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqbc\" (UniqueName: \"kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.434735 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.434735 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.434735 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.434735 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.434735 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.435060 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.435060 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.434776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2f7f\" (UniqueName: \"kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.450147 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.450120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:07.483256 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.483215 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 06:17:07.535936 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.535906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536062 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.535942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536062 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.535960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f7f\" (UniqueName: \"kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536062 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.535980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536062 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.536032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536270 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.536069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.536270 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.536133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.538388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.536658 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.538388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.536892 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.538388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.537787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.540920 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.540902 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:07.541257 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.541166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.541769 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.541542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.542413 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.542388 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.592603 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.592571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f7f\" (UniqueName: \"kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.648423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.648343 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xg9n" event={"ID":"20ebe6ec-1b67-42bd-8f17-873ea5c799a4","Type":"ContainerStarted","Data":"c91f69024a3c8a8180787edbcb685567be93c5e6556634115190fe7feee8ac4e"} Nov 02 06:17:07.648423 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.648415 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xg9n" event={"ID":"20ebe6ec-1b67-42bd-8f17-873ea5c799a4","Type":"ContainerStarted","Data":"e16e8e5f74686c8f5bdc21daeb904a668939600af739999b8186ea5dba7522a3"} Nov 02 06:17:07.654424 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.654353 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f964984-9rxlg" event={"ID":"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52","Type":"ContainerStarted","Data":"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de"} Nov 02 06:17:07.655588 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.655567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:07.773954 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.773908 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7xg9n" podStartSLOduration=3.040663852 podStartE2EDuration="5.773892541s" totalImagesPullingTime="2.646910832s" totalInitContainerRuntime="86.317857ms" isStatefulPod=true podCreationTimestamp="2026-11-02 06:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:07.773088621 +0000 UTC m=+57.021653173" watchObservedRunningTime="2026-11-02 06:17:07.773892541 +0000 UTC m=+57.022457092" Nov 02 06:17:07.844469 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.844431 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-skv9z"] Nov 02 06:17:07.853108 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:07.853075 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b70d77d_9aaa_44d6_8f1b_e68febbd921c.slice/crio-3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e WatchSource:0}: Error finding container 3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e: Status 404 returned error can't find the container with id 3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e Nov 02 06:17:07.888109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:07.888024 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c9f964984-9rxlg" podStartSLOduration=2.359035815 podStartE2EDuration="6.888007958s" totalImagesPullingTime="4.528972143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:07.886942861 +0000 UTC m=+57.135507412" watchObservedRunningTime="2026-11-02 06:17:07.888007958 +0000 UTC m=+57.136572509" Nov 02 06:17:08.030953 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.030921 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64f9df6957-68857"] Nov 02 06:17:08.038672 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:08.038644 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae6eb4c_062d_4118_9bc0_8ac9af0cc881.slice/crio-f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043 WatchSource:0}: Error finding container f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043: Status 404 returned error can't find the container with id f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043 Nov 02 06:17:08.151500 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.151474 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75d975db85-8dlws"] Nov 02 06:17:08.154581 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.154561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.213568 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.213540 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 06:17:08.213664 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.213581 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 06:17:08.214361 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.214334 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 06:17:08.214446 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.214392 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 06:17:08.214446 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.214403 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d58hj\"" Nov 02 06:17:08.214577 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.214558 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 06:17:08.214636 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.214588 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 06:17:08.231241 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.231213 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75d975db85-8dlws"] Nov 02 06:17:08.242223 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242301 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242301 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclbn\" (UniqueName: \"kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242439 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242439 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.242550 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.242535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.343721 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.343849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclbn\" (UniqueName: \"kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.343849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.343849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.343849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.344036 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.344036 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.343949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.344036 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.344008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.345482 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.345161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.345482 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.345338 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.346091 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.346014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.347563 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.347517 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.347642 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.347573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.347642 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.347583 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.347859 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.347834 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.398445 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.398417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclbn\" (UniqueName: \"kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.468821 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.468751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:08.657964 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.657927 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" event={"ID":"7b70d77d-9aaa-44d6-8f1b-e68febbd921c","Type":"ContainerStarted","Data":"3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e"} Nov 02 06:17:08.659629 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.659600 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715" exitCode=0 Nov 02 06:17:08.659731 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.659689 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715"} Nov 02 06:17:08.663507 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.663474 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64f9df6957-68857" event={"ID":"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881","Type":"ContainerStarted","Data":"f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043"} Nov 02 06:17:08.951398 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:08.951318 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75d975db85-8dlws"] Nov 02 06:17:08.959401 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:08.959373 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6df5824_c856_4945_b1da_7027bf549f00.slice/crio-0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a WatchSource:0}: Error finding container 0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a: Status 404 returned error can't find the container with id 0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a Nov 02 06:17:09.677684 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:09.677644 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" event={"ID":"d6df5824-c856-4945-b1da-7027bf549f00","Type":"ContainerStarted","Data":"0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a"} Nov 02 06:17:10.685965 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:10.685684 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"3bbac7704bb2ea0ba40f8ed6b8c7fed26c73fdadc75de8c8d86d9b694cf27828"} Nov 02 06:17:10.685965 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:10.685771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"ccea43f8b74e804390ea4bd9ef4395b0f9dd36b56d1ac0bcaeeb5e41fddb6839"} Nov 02 06:17:10.685965 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:10.685816 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"a6e52286f52d4c0cc287a817dd18e486c9aea2a2931fdbc47bce80cfc60adf18"} Nov 02 06:17:11.691404 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.691367 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" event={"ID":"7b70d77d-9aaa-44d6-8f1b-e68febbd921c","Type":"ContainerStarted","Data":"1610cd246f2a37f8bb549a11c0bbfca80c3de4a072326ffa35a92179231abea1"} Nov 02 06:17:11.692682 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.692634 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:11.699110 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.699088 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:11.700059 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.700036 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6"} Nov 02 06:17:11.700161 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.700086 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca"} Nov 02 06:17:11.702040 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.701982 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64f9df6957-68857" event={"ID":"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881","Type":"ContainerStarted","Data":"6a3edbb412cf5f9669e02408dea058869bd172aed09e6e54e01d5ddb0896ba29"} Nov 02 06:17:11.736833 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.736808 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:11.740918 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.740898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.778969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781048 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781717 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781717 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781717 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.781717 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.779318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.790711 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.790831 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.790855 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.790894 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.790975 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791061 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791245 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791344 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791401 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791496 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cmgkj\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791575 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ddeqh841j2mi8\"" Nov 02 06:17:11.792109 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.791618 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 06:17:11.838746 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.838665 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" podStartSLOduration=1.505117261 podStartE2EDuration="4.838646396s" totalImagesPullingTime="3.333529135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:11.836595944 +0000 UTC m=+61.085160507" watchObservedRunningTime="2026-11-02 06:17:11.838646396 +0000 UTC m=+61.087210947" Nov 02 06:17:11.848925 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.848900 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.881202 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.882321 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.882321 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.882321 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.880749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.882321 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.882170 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.883767 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.883741 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.884307 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.884286 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.885171 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.885144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.889243 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.889216 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.889243 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.889233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.889461 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.889437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.889525 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.889486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.889788 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.889767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.890130 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.890110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.890481 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.890459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.890524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.890497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.890524 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.890507 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.891767 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.891420 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 06:17:11.891767 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.891474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.891767 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.891616 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 06:17:11.892492 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.892465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.896388 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.896348 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.902272 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.902221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.940221 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.940138 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 06:17:11.940348 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.940333 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 06:17:11.943540 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.943516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.943976 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.943956 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:11.948616 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:11.948570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:12.070884 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.070806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:12.201382 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.201322 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64f9df6957-68857" podStartSLOduration=2.255795661 podStartE2EDuration="5.201298742s" totalImagesPullingTime="2.945503081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:12.200309722 +0000 UTC m=+61.448874275" watchObservedRunningTime="2026-11-02 06:17:12.201298742 +0000 UTC m=+61.449863296" Nov 02 06:17:12.310493 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.309862 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:12.310493 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.309932 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:12.315477 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.315451 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:12.523178 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.523151 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:12.529745 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:12.529719 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b1ccb3_168c_43bd_8642_57e6e7b4f430.slice/crio-e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c WatchSource:0}: Error finding container e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c: Status 404 returned error can't find the container with id e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c Nov 02 06:17:12.713782 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.713748 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889"} Nov 02 06:17:12.714198 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.713822 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212"} Nov 02 06:17:12.714198 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.713863 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d"} Nov 02 06:17:12.717516 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.717489 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"36db8f5779b1c2d6fce5a1385e325931ea5760eb9b7061c99338c9075996e1b1"} Nov 02 06:17:12.717633 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.717556 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"623eab921434ed3d536c3134a0c745f1eda1c09e7059689b213b4d640fded721"} Nov 02 06:17:12.717691 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.717635 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:12.717691 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.717676 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" event={"ID":"206c88d3-cdff-496a-ac41-e8e7feca0ac8","Type":"ContainerStarted","Data":"ef4eb7b674b5323e343f1ed69cecaf39da72afe94c90d8f92c33929a8eb24f18"} Nov 02 06:17:12.720659 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.720094 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" event={"ID":"d6df5824-c856-4945-b1da-7027bf549f00","Type":"ContainerStarted","Data":"79ff8d0f5b227318ec71047de689b1d2c036947f8340c4edb82777387b3c0865"} Nov 02 06:17:12.720659 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.720158 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" event={"ID":"d6df5824-c856-4945-b1da-7027bf549f00","Type":"ContainerStarted","Data":"2ccf9fc534645cb3d66e841a3b1f6f7fe7dba9fd9cc9aa45d11c94c8c596875e"} Nov 02 06:17:12.720659 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.720188 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" event={"ID":"d6df5824-c856-4945-b1da-7027bf549f00","Type":"ContainerStarted","Data":"f18c3b1f7f3721b72c246aa16923b222c1accfa239f1f2a4fdc12932c3e1f3f9"} Nov 02 06:17:12.722173 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.722149 2579 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" exitCode=0 Nov 02 06:17:12.722286 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.722247 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} Nov 02 06:17:12.722344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.722313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c"} Nov 02 06:17:12.726720 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.726698 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:12.834308 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:12.834253 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" podStartSLOduration=1.573578863 podStartE2EDuration="7.834232338s" totalImagesPullingTime="6.260653475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:12.832517691 +0000 UTC m=+62.081082242" watchObservedRunningTime="2026-11-02 06:17:12.834232338 +0000 UTC m=+62.082796890" Nov 02 06:17:13.171940 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:13.171882 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" podStartSLOduration=1.864578565 podStartE2EDuration="5.171863713s" totalImagesPullingTime="3.307285148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:13.171696882 +0000 UTC m=+62.420261445" watchObservedRunningTime="2026-11-02 06:17:13.171863713 +0000 UTC m=+62.420428280" Nov 02 06:17:13.731177 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:13.731136 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerStarted","Data":"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2"} Nov 02 06:17:13.850757 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:13.850487 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.055744926 podStartE2EDuration="9.850472265s" totalImagesPullingTime="5.783702254s" totalInitContainerRuntime="11.025085ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:13.848593594 +0000 UTC m=+63.097158181" watchObservedRunningTime="2026-11-02 06:17:13.850472265 +0000 UTC m=+63.099036816" Nov 02 06:17:14.587947 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.587896 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:17:14.591137 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.591107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.642940 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.642905 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 06:17:14.648366 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.648338 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:17:14.716302 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716459 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716459 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716549 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716590 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716590 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716674 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.716724 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.716696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.817893 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.817854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.817916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.817962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.818037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.818121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.818149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.818227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.818377 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.818336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.819742 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.819716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.820217 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.820193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.820328 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.820290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.822353 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.821049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.822353 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.821117 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.822353 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.821804 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.822918 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.822894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.874726 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.874636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:14.905084 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:14.905061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:15.278545 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:15.278508 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:17:15.743330 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:15.743288 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f78595c7-79l6n" event={"ID":"a7c9168b-aa81-46fc-a51a-ec217aa704df","Type":"ContainerStarted","Data":"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b"} Nov 02 06:17:15.743330 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:15.743338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f78595c7-79l6n" event={"ID":"a7c9168b-aa81-46fc-a51a-ec217aa704df","Type":"ContainerStarted","Data":"d52d8b31c3970d70efef4ec4b4fa859b1fa47b54350c830c3332d289b0b52515"} Nov 02 06:17:15.854586 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:15.853357 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58f78595c7-79l6n" podStartSLOduration=1.853340297 podStartE2EDuration="1.853340297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:15.852875406 +0000 UTC m=+65.101439957" watchObservedRunningTime="2026-11-02 06:17:15.853340297 +0000 UTC m=+65.101904847" Nov 02 06:17:16.332692 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.332650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:16.332894 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.332792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:16.381033 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.380958 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:17:16.381033 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.380962 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:17:16.389510 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.389483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:16.393597 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.393575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:16.433875 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.433845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:16.482209 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.482170 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:17:16.531925 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.531888 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:17:16.537316 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.537292 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:16.579849 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.579825 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:17:16.582960 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.582900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:16.589530 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.589504 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:17:16.594274 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.592670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:16.708198 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.708169 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:17:16.711273 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.711249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:16.976903 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:16.976882 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h5h5g"] Nov 02 06:17:16.989079 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:16.989050 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab2c88a_127f_4391_870a_f9757c5f9b04.slice/crio-b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7 WatchSource:0}: Error finding container b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7: Status 404 returned error can't find the container with id b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7 Nov 02 06:17:17.030344 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.030319 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vd4bl"] Nov 02 06:17:17.041176 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:17.041152 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c2f13d_6a20_4247_b926_ad21d58dc644.slice/crio-b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1 WatchSource:0}: Error finding container b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1: Status 404 returned error can't find the container with id b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1 Nov 02 06:17:17.103310 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.103236 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s4zwp"] Nov 02 06:17:17.118072 ip-10-0-142-103 kubenswrapper[2579]: W1102 06:17:17.118042 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ba8ca_d3fa_43e1_b55c_f6217f8a5a69.slice/crio-c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d WatchSource:0}: Error finding container c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d: Status 404 returned error can't find the container with id c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d Nov 02 06:17:17.755486 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755446 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} Nov 02 06:17:17.755685 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755532 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} Nov 02 06:17:17.755685 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755575 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} Nov 02 06:17:17.755685 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755619 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} Nov 02 06:17:17.755685 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755654 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} Nov 02 06:17:17.755882 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.755692 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerStarted","Data":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} Nov 02 06:17:17.757011 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.756956 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4zwp" event={"ID":"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69","Type":"ContainerStarted","Data":"c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d"} Nov 02 06:17:17.758547 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.758514 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vd4bl" event={"ID":"64c2f13d-6a20-4247-b926-ad21d58dc644","Type":"ContainerStarted","Data":"b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1"} Nov 02 06:17:17.759966 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.759946 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h5h5g" event={"ID":"fab2c88a-127f-4391-870a-f9757c5f9b04","Type":"ContainerStarted","Data":"b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7"} Nov 02 06:17:17.919938 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:17.919889 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.701359181 podStartE2EDuration="6.919871613s" totalImagesPullingTime="4.19169085s" totalInitContainerRuntime="26.821582ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:17.91821828 +0000 UTC m=+67.166782871" watchObservedRunningTime="2026-11-02 06:17:17.919871613 +0000 UTC m=+67.168436166" Nov 02 06:17:18.094233 ip-10-0-142-103 kubenswrapper[2579]: E1102 06:17:18.094157 2579 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:17:18.739667 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:18.739637 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:19.772767 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:19.772734 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4zwp" event={"ID":"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69","Type":"ContainerStarted","Data":"c2ad10be6bd8417f240f884bfbc3f30a3e20982ac821b61e390b2a9cd5417d0e"} Nov 02 06:17:20.779491 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.779455 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kkplp" event={"ID":"80b9b535-bd12-4e37-8b22-bf77f7cd93db","Type":"ContainerStarted","Data":"f812ecd2e669755ae431cb5384ac98b047a51c2bb65bfbad39d0696a10453af6"} Nov 02 06:17:20.779875 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.779629 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:20.781966 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.781925 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4zwp" event={"ID":"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69","Type":"ContainerStarted","Data":"6a7f5fda6a58cff9096f722ac0bbd3671196729276ba61e10c92e4982d19dfa8"} Nov 02 06:17:20.783911 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.783887 2579 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-kkplp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body= Nov 02 06:17:20.784037 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.783935 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-kkplp" podUID="80b9b535-bd12-4e37-8b22-bf77f7cd93db" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" Nov 02 06:17:20.886463 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.886421 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kkplp" podStartSLOduration=1.997912803 podStartE2EDuration="27.886408086s" totalImagesPullingTime="25.888495283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:20.885410175 +0000 UTC m=+70.133974726" watchObservedRunningTime="2026-11-02 06:17:20.886408086 +0000 UTC m=+70.134972636" Nov 02 06:17:20.992646 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:20.992586 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s4zwp" podStartSLOduration=67.603496883 podStartE2EDuration="1m9.992566974s" totalImagesPullingTime="2.389070091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:20.991527676 +0000 UTC m=+70.240092229" watchObservedRunningTime="2026-11-02 06:17:20.992566974 +0000 UTC m=+70.241131526" Nov 02 06:17:21.799331 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:21.799275 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:22.071705 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:22.071608 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:23.800873 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:23.800763 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vd4bl" event={"ID":"64c2f13d-6a20-4247-b926-ad21d58dc644","Type":"ContainerStarted","Data":"8cc2b8970aab57319b654325be29f468e9599f16785e827cba393c94bc394cd4"} Nov 02 06:17:23.801557 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:23.801532 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:23.910894 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:23.910835 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vd4bl" podStartSLOduration=66.807424307 podStartE2EDuration="1m12.910817507s" totalImagesPullingTime="6.1033932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:23.908338702 +0000 UTC m=+73.156903255" watchObservedRunningTime="2026-11-02 06:17:23.910817507 +0000 UTC m=+73.159382059" Nov 02 06:17:24.905624 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:24.905587 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:24.906179 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:24.905665 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:24.912227 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:24.912201 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:25.813882 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:25.813836 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h5h5g" event={"ID":"fab2c88a-127f-4391-870a-f9757c5f9b04","Type":"ContainerStarted","Data":"a55e492a3740b719348e604407bdfcfd5f29e3a5fd24108781ad27f97565ce02"} Nov 02 06:17:25.819131 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:25.819071 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:25.923479 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:25.923416 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h5h5g" podStartSLOduration=66.326315739 podStartE2EDuration="1m14.923397524s" totalImagesPullingTime="8.597081785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:25.921609184 +0000 UTC m=+75.170173736" watchObservedRunningTime="2026-11-02 06:17:25.923397524 +0000 UTC m=+75.171962077" Nov 02 06:17:25.938199 ip-10-0-142-103 kubenswrapper[2579]: I1102 06:17:25.938071 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:17:25.938299 ip-10-0-142-103 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 06:17:25.947687 ip-10-0-142-103 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 06:17:25.947874 ip-10-0-142-103 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 06:17:25.948141 ip-10-0-142-103 systemd[1]: kubelet.service: Consumed 5.090s CPU time. Nov 02 06:17:25.957800 ip-10-0-142-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:17:26.069771 ip-10-0-142-103 kubenswrapper[6273]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:26.069771 ip-10-0-142-103 kubenswrapper[6273]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:17:26.069771 ip-10-0-142-103 kubenswrapper[6273]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:26.069771 ip-10-0-142-103 kubenswrapper[6273]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:26.072181 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072084 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.072181 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072178 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.072181 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072185 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072191 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072196 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072201 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072205 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072208 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072211 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072214 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072217 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072220 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072223 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072225 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072228 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072232 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072237 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072241 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072245 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072250 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072253 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.072380 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072256 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072258 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072262 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072266 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072271 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072285 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072289 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072294 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072297 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072300 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072304 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072308 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072313 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072317 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072321 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072325 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072329 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072333 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072338 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072343 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.073011 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072347 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072352 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072356 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072360 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072364 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072369 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072373 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072376 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072379 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072382 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072385 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072388 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072390 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072393 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072396 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072400 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072402 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072406 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072410 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.073670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072415 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072419 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072423 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072426 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072428 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072431 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072434 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072436 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072439 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072442 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072445 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072447 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072450 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072452 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072456 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072458 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072461 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072464 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072466 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.074366 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072469 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072472 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072476 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072481 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072485 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072488 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072490 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072493 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072497 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072499 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072502 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072505 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072509 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072514 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072518 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072522 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072526 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072530 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072534 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.075062 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072539 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072543 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072547 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072551 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072556 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072560 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072566 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.072569 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073579 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073590 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073595 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073600 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073604 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073608 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073615 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073623 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073627 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073631 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073634 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073639 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.075678 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073643 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073648 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073653 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073658 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073662 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073666 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073676 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073680 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073683 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073686 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073689 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073691 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073694 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073697 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073699 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073702 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073705 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073708 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073710 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.076403 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073713 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073715 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073718 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073720 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073723 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073726 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073728 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073731 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073736 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073738 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073741 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073743 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073746 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073749 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073752 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073754 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073757 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073760 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073764 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073769 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.077106 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073774 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073778 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073783 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073788 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073792 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073797 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073801 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073805 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073810 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073814 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073820 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073824 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073827 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073830 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073832 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073835 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073838 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073840 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073843 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073846 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.077889 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073852 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073856 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073860 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073863 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073866 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073869 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073872 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073874 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073877 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073880 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073883 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073885 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073888 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073891 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073894 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073897 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073899 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073902 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073905 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.078574 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073907 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073910 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073913 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073916 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073918 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073921 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073923 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073926 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073930 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073936 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073941 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073945 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073950 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073956 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073961 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.073965 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074561 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074570 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074574 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.079232 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074577 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074580 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074582 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074585 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074588 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074590 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074593 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074596 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074599 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074602 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074604 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074607 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074611 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074615 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074619 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074624 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074628 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074632 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074636 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.079933 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074641 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074643 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074647 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074651 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074655 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074659 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074664 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074672 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074677 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074681 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074684 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074687 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074690 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074693 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074697 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074702 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074706 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074709 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074711 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.080635 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074714 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074717 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074720 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074723 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074725 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074728 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074732 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074737 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074741 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074745 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074749 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074753 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074758 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074763 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074767 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074771 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074776 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074782 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074786 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074791 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.081292 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074795 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074800 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074805 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074809 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074814 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074820 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074826 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074831 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074836 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074840 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074844 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074848 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074852 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074856 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074861 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074866 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074870 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074874 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074878 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.082028 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074883 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074887 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074891 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074894 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074899 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074904 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074908 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074912 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074916 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074921 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074925 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074929 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074934 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074940 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074945 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074948 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074950 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074953 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074956 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074958 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.082846 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074961 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074965 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074969 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074973 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074977 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.074981 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075114 6273 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075124 6273 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075130 6273 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075136 6273 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075141 6273 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075144 6273 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075151 6273 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075156 6273 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075159 6273 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075162 6273 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075166 6273 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075169 6273 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075172 6273 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075175 6273 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075178 6273 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075182 6273 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075186 6273 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:17:26.083696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075190 6273 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075198 6273 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075205 6273 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075208 6273 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075212 6273 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075215 6273 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075219 6273 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075223 6273 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075227 6273 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075232 6273 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075236 6273 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075241 6273 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075246 6273 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075251 6273 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075254 6273 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075259 6273 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075262 6273 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075265 6273 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075268 6273 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075272 6273 flags.go:64] FLAG: --enable-server="true" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075275 6273 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075280 6273 flags.go:64] FLAG: --event-burst="100" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075284 6273 flags.go:64] FLAG: --event-qps="50" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075287 6273 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:17:26.084714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075290 6273 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075293 6273 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075297 6273 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075300 6273 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075303 6273 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075306 6273 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075309 6273 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075312 6273 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075315 6273 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075318 6273 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075321 6273 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075324 6273 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075327 6273 flags.go:64] FLAG: --feature-gates="" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075335 6273 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075338 6273 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075341 6273 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075344 6273 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075348 6273 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075351 6273 flags.go:64] FLAG: --help="false" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075354 6273 flags.go:64] FLAG: --hostname-override="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075357 6273 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075362 6273 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075367 6273 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:17:26.085862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075373 6273 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075377 6273 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075380 6273 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075383 6273 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075385 6273 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075389 6273 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075394 6273 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075397 6273 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075400 6273 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075426 6273 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075431 6273 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075437 6273 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075441 6273 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075447 6273 flags.go:64] FLAG: --lock-file="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075451 6273 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075456 6273 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075461 6273 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075470 6273 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075474 6273 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075480 6273 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075485 6273 flags.go:64] FLAG: --logging-format="text" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075489 6273 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075495 6273 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075498 6273 flags.go:64] FLAG: --manifest-url="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075501 6273 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:17:26.086902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075506 6273 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075509 6273 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075513 6273 flags.go:64] FLAG: --max-pods="110" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075516 6273 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075520 6273 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075523 6273 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075526 6273 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075529 6273 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075534 6273 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075537 6273 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075544 6273 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075549 6273 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075555 6273 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075562 6273 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075565 6273 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075568 6273 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075572 6273 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075575 6273 flags.go:64] FLAG: --port="10250" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075578 6273 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075581 6273 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01b93421fa03ee9b5" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075584 6273 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075587 6273 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075590 6273 flags.go:64] FLAG: --register-node="true" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075595 6273 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075601 6273 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075606 6273 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:17:26.088116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075610 6273 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075615 6273 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075621 6273 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075626 6273 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075631 6273 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075636 6273 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075641 6273 flags.go:64] FLAG: --runonce="false" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075646 6273 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075651 6273 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075655 6273 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075660 6273 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075665 6273 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075671 6273 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075674 6273 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075680 6273 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075685 6273 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075690 6273 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075695 6273 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075700 6273 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075707 6273 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075713 6273 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075718 6273 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075725 6273 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075728 6273 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075731 6273 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:17:26.089372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075736 6273 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075739 6273 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075742 6273 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075745 6273 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075749 6273 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075752 6273 flags.go:64] FLAG: --v="2" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075756 6273 flags.go:64] FLAG: --version="false" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075761 6273 flags.go:64] FLAG: --vmodule="" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075766 6273 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.075772 6273 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075940 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075945 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075948 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075951 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075954 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075956 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075959 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075962 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075965 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075968 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075970 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075975 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.090424 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075978 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075980 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075983 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.075998 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076003 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076006 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076009 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076012 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076015 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076018 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076021 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076025 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076030 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076034 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076038 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076043 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076047 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076051 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.091330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076056 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076062 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076068 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076072 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076077 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076082 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076086 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076090 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076092 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076097 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076101 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076104 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076107 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076112 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076116 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076120 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076124 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076128 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076134 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.091961 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076137 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076140 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076143 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076146 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076148 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076151 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076154 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076157 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076159 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076162 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076165 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076168 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076170 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076173 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076176 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076178 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076181 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076184 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076186 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076189 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.092604 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076191 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076194 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076196 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076199 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076202 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076204 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076209 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076212 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076214 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076217 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076221 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076227 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076232 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076236 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076240 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076246 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076250 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076254 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076259 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076263 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076268 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.093388 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076272 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076276 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076280 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076285 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076289 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076294 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076298 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076303 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076308 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076312 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076316 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076319 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076322 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076325 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076328 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.094193 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076331 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.076368 6273 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076424 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076431 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076434 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076437 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076440 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076444 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076447 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076450 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076453 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076456 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076460 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076463 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076466 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.094855 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076468 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076471 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076474 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076476 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076480 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076487 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076493 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076498 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076503 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076507 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076511 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076515 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076520 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076524 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076528 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076531 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076534 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076537 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076540 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.095452 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076542 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076546 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076549 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076559 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076562 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076568 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076572 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076577 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076581 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076586 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076590 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076594 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076598 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076601 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076603 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076606 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076608 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076611 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076614 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076617 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.096136 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076620 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076623 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076625 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076628 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076630 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076633 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076635 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076638 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076641 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076643 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076646 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076649 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076651 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076655 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076658 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076663 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076667 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076674 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076679 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076683 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.096727 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076687 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076691 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076695 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076700 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076705 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076709 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076714 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076718 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076722 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076727 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076731 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076735 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076740 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076744 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076748 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076752 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076756 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076760 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076765 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.097418 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076769 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076773 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076777 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076781 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076786 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076790 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076796 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076800 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076811 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076816 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076822 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076827 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076832 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076836 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.076840 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.082594 6273 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.082615 6273 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.082640 6273 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.082648 6273 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:17:26.098086 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082809 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082818 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082824 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082829 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082834 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082839 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082843 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082848 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082852 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082862 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082866 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082871 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082875 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082880 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082884 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082889 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082893 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082897 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082903 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.098724 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082907 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082912 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082921 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082926 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082930 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082935 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082939 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082943 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082948 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082952 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082958 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082962 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082967 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082971 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.082975 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083000 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083005 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083010 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083014 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083018 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.099379 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083023 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083027 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083031 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083036 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083041 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083045 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083049 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083053 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083062 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083066 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083071 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083075 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083080 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083084 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083089 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083093 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083097 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083102 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083106 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.100019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083110 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083121 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083127 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083132 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083137 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083140 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083143 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083146 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083148 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083151 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083154 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083159 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083164 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083173 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083178 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083182 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083187 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083192 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083197 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.100691 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083201 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083206 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083210 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083214 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083219 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083224 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083228 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083237 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083242 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083246 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083250 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083255 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083259 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083263 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083267 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083271 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083276 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083280 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083285 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.101371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083295 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083299 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083304 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083308 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083312 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083317 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083322 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083326 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083330 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083335 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.083395 6273 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.083974 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084005 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084013 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084017 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084022 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.102199 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084026 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084031 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084035 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084040 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084044 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084049 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084053 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084057 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084061 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084066 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084070 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084075 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084080 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084084 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084088 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084092 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084098 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084102 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084107 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.102707 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084111 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084116 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084121 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084126 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084130 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084137 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084142 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084146 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084151 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084155 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084160 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084164 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084168 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084173 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084177 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084181 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084185 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084190 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084195 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084199 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.103359 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084203 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084208 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084212 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084216 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084220 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084227 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084234 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084239 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084243 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084248 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084252 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084256 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084260 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084265 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084269 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084274 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084278 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084285 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084290 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084294 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.104146 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084298 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084304 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084311 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084315 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084319 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084324 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084328 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084333 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084337 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084342 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084346 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084350 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084355 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084359 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084363 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084367 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084372 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084376 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084380 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.104835 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084385 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084389 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084393 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084398 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084402 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084406 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084410 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084415 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084420 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084424 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084430 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084435 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084439 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084443 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084447 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084452 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084456 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084460 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084464 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.105556 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084469 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084473 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084479 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084483 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.084526 6273 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084581 6273 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084587 6273 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084592 6273 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084597 6273 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084602 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084608 6273 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084614 6273 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084619 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084624 6273 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084628 6273 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084633 6273 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:26.106285 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084637 6273 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084641 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084646 6273 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084650 6273 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084656 6273 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084661 6273 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084665 6273 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084669 6273 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084677 6273 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084681 6273 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084688 6273 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084694 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084699 6273 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084703 6273 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084708 6273 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084711 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084716 6273 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084720 6273 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084724 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084730 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:26.106851 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084734 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084738 6273 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084742 6273 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084746 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084751 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084756 6273 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084760 6273 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084765 6273 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084769 6273 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084773 6273 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084777 6273 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084781 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084786 6273 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084790 6273 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084794 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084799 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084813 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084819 6273 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084823 6273 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:26.107599 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084828 6273 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084832 6273 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084837 6273 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084841 6273 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084845 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084849 6273 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084853 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084858 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084862 6273 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084866 6273 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084871 6273 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084875 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084880 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084885 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084889 6273 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084894 6273 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084898 6273 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084902 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084908 6273 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:26.108431 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084912 6273 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084916 6273 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084921 6273 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084925 6273 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084929 6273 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084933 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084937 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084941 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084946 6273 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084951 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084957 6273 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084961 6273 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084965 6273 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084970 6273 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084974 6273 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084978 6273 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.084982 6273 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085011 6273 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085017 6273 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:26.109263 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085022 6273 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085026 6273 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085030 6273 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085034 6273 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085039 6273 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085043 6273 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085047 6273 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085051 6273 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085057 6273 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085061 6273 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085067 6273 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085072 6273 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085076 6273 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085080 6273 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085084 6273 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085088 6273 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085092 6273 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:26.085097 6273 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.085181 6273 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.086691 6273 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 06:17:26.110221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.086755 6273 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.087033 6273 server.go:1014] "Starting client certificate rotation" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.087179 6273 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.087239 6273 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:11:11 +0000 UTC" deadline="2028-06-22 08:08:14.171837603 +0000 UTC" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.087276 6273 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14353h50m48.084564187s" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.087721 6273 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.089460 6273 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.091000 6273 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.093557 6273 log.go:25] "Validated CRI v1 image API" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.094581 6273 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:17:26.111073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.097820 6273 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 805c9131-05aa-4245-b502-f7e0bce13636:/dev/nvme0n1p3 c9e024ef-8c1d-40c8-bee7-12f99b722eb0:/dev/nvme0n1p4] Nov 02 06:17:26.111562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.097845 6273 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1/userdata/shm major:0 minor:509 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7/userdata/shm major:0 minor:507 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d/userdata/shm major:0 minor:511 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d52d8b31c3970d70efef4ec4b4fa859b1fa47b54350c830c3332d289b0b52515/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d52d8b31c3970d70efef4ec4b4fa859b1fa47b54350c830c3332d289b0b52515/userdata/shm major:0 minor:494 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c/userdata/shm major:0 minor:451 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb289fe-8e1a-4abe-a37b-4cc6ac343150/volumes/kubernetes.io~projected/kube-api-access-d2fg8:{mountpoint:/var/lib/kubelet/pods/0eb289fe-8e1a-4abe-a37b-4cc6ac343150/volumes/kubernetes.io~projected/kube-api-access-d2fg8 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/alertmanager/9 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/config-reloader/6 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/kube-api-access-jhqxv:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/kube-api-access-jhqxv major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/tls-assets major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/config-volume major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/web-config major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~projected/kube-api-access-9wgkr:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~projected/kube-api-access-9wgkr major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~projected/kube-api-access-n6p26:{mountpoint:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~projected/kube-api-access-n6p26 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2abb23a4-08db-4159-8715-df445adc7a71/volumes/kubernetes.io~projected/kube-api-access-zdlv2:{mountpoint:/var/lib/kubelet/pods/2abb23a4-08db-4159-8715-df445adc7a71/volumes/kubernetes.io~projected/kube-api-access-zdlv2 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~projected/kube-api-access-b2f7f:{mountpoint:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~projected/kube-api-access-b2f7f major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~projected/kube-api-access-f7pkr:{mountpoint:/var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~projected/kube-api-access-f7pkr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~projected/kube-api-access-h6q8x:{mountpoint:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~projected/kube-api-access-h6q8x major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37012b0f-1510-431f-9811-f97e41935c40/volumes/kubernetes.io~projected/kube-api-access-jzv2m:{mountpoint:/var/lib/kubelet/pods/37012b0f-1510-431f-9811-f97e41935c40/volumes/kubernetes.io~projected/kube-api-access-jzv2m major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~projected/kube-api-access-qnkxk:{mountpoint:/var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~projected/kube-api-access-qnkxk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64c2f13d-6a20-4247-b926-ad21d58dc644/volumes/kubernetes.io~projected/kube-api-access-brvn6:{mountpoint:/var/lib/kubelet/pods/64c2f13d-6a20-4247-b926-ad21d58dc644/volumes/kubernetes.io~projected/kube-api-access-brvn6 major:0 minor:506 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8/volumes/kubernetes.io~secret/tls-certificates major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~projected/kube-api-access-cdqbc:{mountpoint:/var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~projected/kube-api-access-cdqbc major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b9b535-bd12-4e37-8b22-bf77f7cd93db/volumes/kubernetes.io~projected/kube-api-access-jzs2k:{mountpoint:/var/lib/kubelet/pods/80b9b535-bd12-4e37-8b22-bf77f7cd93db/volumes/kubernetes.io~projected/kube-api-access-jzs2k major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853c435f-0786-450f-bdd9-f95cc551ba78/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/853c435f-0786-450f-bdd9-f95cc551ba78/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~projected/kube-api-access-r7s8k:{mountpoint:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~projected/kube-api-access-r7s8k major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:489 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:485 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~projected/kube-api-access-h7fzk:{mountpoint:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~projected/kube-api-access-h7fzk major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~projected/kube-api-access-dblwh:{mountpoint:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~projected/kube-api-access-dblwh major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~projected/kube-api-access-rbwx9:{mountpoint:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~projected/kube-api-access-rbwx9 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~projected/kube-api-access-tclbn:{mountpoint:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~projected/kube-api-access-tclbn major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~projected/kube-api-access-rdkbd:{mountpoint:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~projected/kube-api-access-rdkbd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~projected/kube-api-access-trqwx:{mountpoint:/var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~projected/kube-api-access-trqwx major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~secret/metrics-certs major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/config-reloader/2 major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/prometheus/15 major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~empty-dir/config-out major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/kube-api-access-9jx64:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/kube-api-access-9jx64 major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/tls-assets major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/config major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/web-config major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fab2c88a-127f-4391-870a-f9757c5f9b04/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fab2c88a-127f-4391-870a-f9757c5f9b04/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:505 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd25d9ad-68e0-49c8-a9cf-0ca18a976680/volumes/kubernetes.io~projected/kube-api-access-h2xmh:{mountpoint:/var/lib/kubelet/pods/fd25d9ad-68e0-49c8-a9cf-0ca18a976680/volumes/kubernetes.io~projected/kube-api-access-h2xmh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe8c4326-8500-4a72-848f-7281b21b4853/volumes/kubernetes.io~projected/kube-api-access-5xpxt:{mountpoint:/var/lib/kubelet/pods/fe8c4326-8500-4a72-848f-7281b21b4853/volumes/kubernetes.io~projected/kube-api-access-5xpxt major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~projected/kube-api-access-j6fx8:{mountpoint:/var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~projected/kube-api-access-j6fx8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff9ab1b6-621a-430c-9669-1071af47a675/volumes/kubernetes.io~projected/kube-api-access-chtbc:{mountpoint:/var/lib/kubelet/pods/ff9ab1b6-621a-430c-9669-1071af47a675/volumes/kubernetes.io~projected/kube-api-access-chtbc major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fdd43411fb2839d8efcba8582cfe877fe3207c57e8720f8977e25c986c4e9e7d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/68a7c84bd778abc245a375c98c8043df63e3a264280caf413bb13b593ddbdd5f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b4cbe4a5aa23bd6d1de211ed29f6e16089cd0784038b865e81a07f0211b8c352/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5090af478c0dfb1cc33a4ffbf51c250a0af500ff90696fc5da39a64317dd1cba/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d9b9acc440750b3f83c2c2307b15bcb6ebdbe2c1d1c1fa0faff90b20cccaac8d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9a9a08cc7a74dada8e2bda76af1229276f1d2240097da536308018ff356dfaa0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4358cf541041c9570ef5121db4c1978adac59c2173a3ca484fa47c02ce88e650/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7cf189577b33f1da96f4c5cf38fd80798433a1d8f1d24841c147afd56c6a0357/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/68636ec685ccdc672f0083bcd32d81d3a0fbfd6bfd94c614b4d73ead4a68ca39/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/acaeb0d20b29cc11c2c99eaa3c002248cd285e6ec849bf60c80850ccc168776e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/c46f046a5fd93da801f53874edb6551e7eee6643a6397a8585311008d4377643/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0cfa6040be60285daf032d637bbfcf56b9f7fc4adbd712528fa707b9696e3763/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/fd9dea7d055b286b9087a071940f132f9bc5082dbab0c5b751ffe5d635dffa3e/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/d81acaa13eb46f66ca59b4d1c1a1c286b5fc2cb17ab1cfe04aa7843861448345/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/204d78225e9ee117e75113bbf97d9382cb0df6c5d7995d3fb224c9c2168a7b0e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7d681b70e128554839c95bdd230662dc78d4e0e4c164f9a3e1f0de7c99d34356/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cb916a91d88d2f20431e882c46313c4c7f096dc6ddc9711e3e071629004b260b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/291b2725b97dd68d37add5dd5d347932b88e42b4e9c376effd912aae89ac0430/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/88fdb49e1788523815fe83c84abeebca622317b16752c2ea00fdf2c2652fe583/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cc2225ff2ec7949acbe4fc5ef752b558940da6fa324d53dd9d62fc83dd9404ba/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-196:{mountpoint:/var/lib/containers/storage/overlay/ccdbe908f9165ba2317036e5032b135620e6b13546c9a58a61c291993eafbf11/merged major:0 minor:196 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c7883ff5cc4e52c5c22cd669901036b9ae98b312d696d969428a6d2c8d06726b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3d9d2d5270f6f9acebfdb4c01296d7972fd9e4abdb399af08f22be822ebb9e6e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/de14315a9258805fddc83f31a14e416827f032ed3763ff606102e9014340c3a7/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d5a3a40eaa56acc3ba189df25513a6a36c101de0e800f55dd57f4c0581bec814/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/aba362358435ce707f181936495abfc57828cebd9a6e52e9cb4262a75261f1b9/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/a4215f12dd0f2ab22c7758d080e2fc088bdeb5fe1b5b824271efa1a3cd951b15/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/0b6de28718e67edcf9800cc220df63b0cc74a8f3f5b2f2a831fd24315e3cc43c/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/632ffd6002c0ea3735f1a0da8385c7d34bceb2dbec32f282071823fc3cabef1c/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9cdd64254c221c37d73e8d151aae677606287fef725103a078f5677a98373a2f/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/265d5b1716543232fddae960e5e46c8ec11394a05c8c420ad16b3ce1a0a5d88a/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/d3f157caa24a0389fdec5efe8fc411e15bd7c84d03703290541866f2461f9574/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/33898d57b63c32d17c1735911c01b48fa5ba648ea4514764be0a730d66aa9516/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d4e6a1a7a18b8d77ab0c79601e4724d2ddcec856e6d4a5221ce6457dc7f5ef2f/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d6ad06f54104aec6d586afa70017b5b1c54e29f33fe2fec144a451860e5e3f7c/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/c5ff8db6bd0567bac7d083ed226c6962d69a2c9ff10cf517c2a46b24464af362/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/5ba522dc168d1d29a8b8ebb75a0077dd26a5095f23fd1f6640c09f3f2c311fe4/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/931e614ef538db5b0590e5a88921c5a353f3d2d493c4e4272d87f4020f2e586b/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/3b8af7a308d12505ae13a9aa8f24bee7cbdee37e7018eda546c6f83b8ba37fc2/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/b0e3a606dce122fcb2976ae1e3814fca88cd3b69bbaa443b91055507a48d5706/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/36b05b1e4817244799805e50c92e6bf9023e8fc84bcc7fb939a48df9de932e02/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/7a997034f07eb491877f3bf6d4948dd22dc8db833841ffd680f97346e39e6916/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f18447d4d914546484f6078725602902167446e85b3337817d775958cc2cb50f/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/f1b7f537b1130e28daa27141f3fecfc5a327ed3adcc222305833e2ae737cf308/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ee3ba2617d18d8c15f0fee88013e9f9cada5a9a4e442e73843ede1088ea0dd55/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/9a67dccbd88c25f3b2038dfe5d46a5560ebbd567540efdf01129884518c8f84c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/66574ec207b327528d758190851ec6b0b77257ef76679d923475344a226ec681/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/f8dc9315813c7c23e34d1a95c5988da88e9b8fc259b0f9c514bbaef81263673f/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/bb78b53caef9d6b761fb0fabe03ba6bb79c65d5e8ff9ec9e875821f010badcf1/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/a44b614b58a978bd678e930ade539cb2833c27cac74e804292854b57ee1c7959/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/8a252a1fe89eb0eeb9988bf407d641241416910108d4667497964164cf71946a/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/83334abff9343a851efddc07562430a4f3c3988674a7e24d42a3c97a216c386b/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/6065d16e767953d6ba0c56326af17dbdda516ee61e5829cb7857c83486d1f2d2/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/cf56068f327b0a69c558f15d185a6fd45ca99ebe3264123597bd28b0a505a5a0/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/427a16bf1599c4279277d1a2a673ee0238f11f704a7aa3ccd69e929736c1f55f/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/46b8178e368ad3992f28931698a4fd63cb25b6538dfdfd0abab2b26512c6ffba/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/f43b5b59cd9633b960d6646449f9ecfce23088fba04b68209d168bf4980460b5/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/698da20f5d887f7facd0484608463403050c1987e6e380ce2326ac4741c7d3fe/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/5cafedd67dafbf616c4af00e41180bf0489f4596acb45681ff4bb982f0ebf421/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/f9cb208da3b54edcf8d93f91b2c123c8b00ff86ca3e03f13a336ed1289c913af/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/f18f27c6f4b3b05f4b18614732d365c7cac270191d0a61653381279f31f929eb/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/97a74ece14274ac54e4f3118f63e30db0a0acdab4c2dc0ce1fc369bafd4a5cf5/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/d132e90d15225cf6bc59a486294cd77c093f5cd3f995962efccca52fab7cea5c/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/22e6a02219cfc58af4d6f076063362906e27003ff41a434466a28205485bc349/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/4f4dec71e2ba9d4320aca3ac6ce7093b01348a78ecca22da9d2780f9ea61aca9/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/2b5895d5e2ffc38cbcbf35877d1d3d06601c85f6786c64c6729acc90cccc8d5b/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/97c7ad4e1a18bc4ff201c854a327094979e4b774dd1b45fc347d8bd8e9f10732/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/cd5c9f0553031733b9038d60724d2f166dae95d1c835918ca7da182866f8e1a6/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/a531de8866fa0de0786efc50846853f139c5b00c43667136ba0e64afff380c5a/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/142e672f428ca03c799af89d4af0b5545651fb4c936c2e848c135ca4dd558463/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/fa5401a2d802c49371149adf20a0a814c71a2e89a8dab6e76c7cdd2349962513/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/b4c807e20efad20e8cfa7bb769a993a7cfc1b535d4d04e6c5124469b06e53bf6/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/627e9359e93dc7ad6b82e816b47ea7808f594669805b812a5bf494da141337a5/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/e706a9f33a6f8bd1885174ebde6461b8546ba113fcb0800446c73f52818e89eb/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/3f4cc4b60c82eaa293e8dc2dd17be8c7f0c09d09db77ca278fcab33cbeb1a5db/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-530:{mountpoint:/var/lib/containers/storage/overlay/ca0d944adb8908de4ec8927fed65a64e84ede3a409003f238b0c70478217e2a3/merged major:0 minor:530 fsType:overlay blockSize:0} overlay_0-532:{mountpoint:/var/lib/containers/storage/overlay/0677f7ed79e5f92b242f1af2bcb714128da3d5fbb0aa7ae509d9100ba503254c/merged major:0 minor:532 fsType:overlay blockSize:0} overlay_0-537:{mountpoint:/var/lib/containers/storage/overlay/0cd1739e8d56c936568e5711e2921a62aa59a8f1a25347767446f1617e414e80/merged major:0 minor:537 fsType:overlay blockSize:0} overlay_0-542:{mountpoint:/var/lib/containers/storage/overlay/b851ce3bcbc97791c077fe0aa64b7219b0f9df371270ef2a8957785cc0461979/merged major:0 minor:542 fsType:overlay blockSize:0} overlay_0-547:{mountpoint:/var/lib/containers/storage/overlay/265180c43fa9fe05e05ecb3e1c3968c215cabf35aea39f534c87362a05dff6d5/merged major:0 minor:547 fsType:overlay blockSize:0} overlay_0-552:{mountpoint:/var/lib/containers/storage/overlay/a95b2a84a159e4d2c504e54189683dee87d77e0d5a216ab53baa325246410cfa/merged major:0 minor:552 fsType:overlay blockSize:0} overlay_0-558:{mountpoint:/var/lib/containers/storage/overlay/848adb14f0a33f651e63bf36bf3032d56ee4e89f4b6a39ddba2e01fb27809657/merged major:0 minor:558 fsType:overlay blockSize:0} overlay_0-563:{mountpoint:/var/lib/containers/storage/overlay/ac7c2959eb7ab7eebef88c4d1b89d2549da2888de1a3d5ada83d9af46165ed42/merged major:0 minor:563 fsType:overlay blockSize:0} overlay_0-569:{mountpoint:/var/lib/containers/storage/overlay/1ebc65a4a487662563b2f145275ad2ac1890e2444c05d35980e74952d2144e81/merged major:0 minor:569 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/45fed08d5d3c1bb9eb03db29f8956beed489bca76f9fc2bddc530d898083de3f/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5a96f2176b330f9369f76400c2d7ee621ac85874057c5f0a2786003b79a72726/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a833acb2e8f2a3c45edd481980a2134c2d9c77ceb70cbce2e27037babc38d319/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mo Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: untpoint:/var/lib/containers/storage/overlay/0d403dd10ef2dfab622e4f516f4b95a1acc3b67509317f6f33583dcf7d013518/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/4b29af14e116429225901f50fbf8c5d6ab6de9aa8be5989206a8a63ca1dd013f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f66074e9b6458d8acb443e779c7228b2576f1ad2f68acb30df0374af13999750/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/777c0d69e4d385e203dce016d6992deb5b131f8deac94dbb386dc3c0472a53d2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0b05c984f4334e576b35bc56b8d847a056afe7ef297826b6227907fa792b6f75/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fbaf084fdd94726cccfa1a58afa4e666bf9ff8d63c34e88dc3644a1f0295076e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/303ddb867d6da5337948c11117e765c27ae93a9b6b62b070ccf15a0d8899d516/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.109677 6273 manager.go:217] Machine: {Timestamp:2026-11-02 06:17:26.108816723 +0000 UTC m=+0.082151400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099879 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2244606ee1de3a7cd73f53ebb7c48b SystemUUID:ec224460-6ee1-de3a-7cd7-3f53ebb7c48b BootID:623e4f3a-5e79-4199-a275-c899378c54a4 Filesystems:[{Device:/run/containers/storage/overlay-containers/e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c/userdata/shm DeviceMajor:0 DeviceMinor:451 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d52d8b31c3970d70efef4ec4b4fa859b1fa47b54350c830c3332d289b0b52515/userdata/shm DeviceMajor:0 DeviceMinor:494 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b463ce8c2b2837e7003038b6ed5f753d76724b3998967392806149ac20296d/userdata/shm DeviceMajor:0 DeviceMinor:511 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-537 DeviceMajor:0 DeviceMinor:537 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-569 DeviceMajor:0 DeviceMinor:569 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37012b0f-1510-431f-9811-f97e41935c40/volumes/kubernetes.io~projected/kube-api-access-jzv2m DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~projected/kube-api-access-9wgkr DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e45eb082846130203a4828c16b5fa6e5d581e8f23e43e8a70aa34724c40c0dbb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ab081a13476dd2cc90ad2c03e48d8880ed7fce50394cca2bcd70121f4df2589/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbcab4d1329ae86f556f7ae339387d226c9d9c97504e17e61c418fa5e1dbd30e/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/kube-api-access-jhqxv DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~projected/kube-api-access-r7s8k DeviceMajor:0 DeviceMinor:490 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd25d9ad-68e0-49c8-a9cf-0ca18a976680/volumes/kubernetes.io~projected/kube-api-access-h2xmh DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-196 DeviceMajor:0 DeviceMinor:196 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:485 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~projected/kube-api-access-h6q8x DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~projected/kube-api-access-n6p26 DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/kube-api-access-9jx64 DeviceMajor:0 DeviceMinor:436 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b86fef7e87926cbaf1378efd70d6b8583ec34e81db742350237ffaa3e0fe41d7/userdata/shm DeviceMajor:0 DeviceMinor:507 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-558 DeviceMajor:0 DeviceMinor:558 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-530 DeviceMajor:0 DeviceMinor:530 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-552 DeviceMajor:0 DeviceMinor:552 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87a6295ea1cdf98bab363b9fd985f4aa4989100b92e66d4b0c7fe0871a12097f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~projected/kube-api-access-rbwx9 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~projected/kube-api-access-j6fx8 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed9041b12c94810783ca182fd7c908117923536e6759ee8374b677b73662600d/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~projected/kube-api-access-cdqbc DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~projected/kube-api-access-b2f7f DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:446 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:500 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/853c435f-0786-450f-bdd9-f95cc551ba78/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe97868d-2a06-4ec7-a874-d88a2dfd0978/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:437 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b70d77d-9aaa-44d6-8f1b-e68febbd921c/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-563 DeviceMajor:0 DeviceMinor:563 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~projected/kube-api-access-rdkbd DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:378 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe8c4326-8500-4a72-848f-7281b21b4853/volumes/kubernetes.io~projected/kube-api-access-5xpxt DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d501b0fe142e88dc20f6a753e633455dca2e38179a3688f66e3dd5d8a6d9c6e/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:379 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:445 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fab2c88a-127f-4391-870a-f9757c5f9b04/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:505 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb289fe-8e1a-4abe-a37b-4cc6ac343150/volumes/kubernetes.io~projected/kube-api-access-d2fg8 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4acd952ae93fe5e33e902158eaf0e3e3383764cbd0e5585b656a460ea834107d/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-547 DeviceMajor:0 DeviceMinor:547 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c93a610c1310cd76700df2a11a7d9f1612929ef8cb8c31247d6935f60f3e9fd4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6c14f265149f7e064c31da2db9c1d8fb14b3f1c2192df65dbb04d62ffd86b99/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85257a07132006eb49f9a74af957c079051c79c8888de02c11de23a52c4ea23c/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:376 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:439 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64c2f13d-6a20-4247-b926-ad21d58dc644/volumes/kubernetes.io~projected/kube-api-access-brvn6 DeviceMajor:0 DeviceMinor:506 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b025b25eea78f0317f194aa2e310315fca0516bcf03607acd6fc43898be046f1/userdata/shm DeviceMajor:0 DeviceMinor:509 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:281 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b424524e-6361-4ccb-9de6-58c78d4021fa/volumes/kubernetes.io~projected/kube-api-access-h7fzk DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:441 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58618324-800f-4705-87d8-d68e27813002/volumes/kubernetes.io~projected/kube-api-access-qnkxk DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/206c88d3-cdff-496a-ac41-e8e7feca0ac8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6ee339ffa94ba5b131b46895cd9423fb08c63b1a3428afdfc021f4f532ee0e4/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20ebe6ec-1b67-42bd-8f17-873ea5c799a4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0d4acf00d9ad7e2aa302838fc519026b9f462b9621be9386f0b8d086bf0175e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~projected/kube-api-access-dblwh DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-532 DeviceMajor:0 DeviceMinor:532 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-542 DeviceMajor:0 DeviceMinor:542 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/daa211aa-bb54-4375-bb65-793ae2e67721/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff9ab1b6-621a-430c-9669-1071af47a675/volumes/kubernetes.io~projected/kube-api-access-chtbc DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/497292758d52fdfd23521251755b9f9b407667286095b0cba422b36a1a84ac2b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:489 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73081fb542a9c7779392824eaacbdfc3225b8e745397893980473b817dd8c902/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~projected/kube-api-access-tclbn DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:440 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05bb480bab7d0efc375f07977ff0fdeb2983e6940fc4712510ff724a62d1787c/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:443 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:447 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2abb23a4-08db-4159-8715-df445adc7a71/volumes/kubernetes.io~projected/kube-api-access-zdlv2 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31af9124013530176456ffcae9e55131a923999106a53e5302c8e87b258d9a44/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9f5321cbae3ae898b5fa76e7b8138d5dfb3c6d455e6c5616615df6e1b0ae043/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c90e67867d10becc8707b17c7accdb1e05fa66fc58a30c6ce2abf5723d88ab93/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba5c9a49-a662-4b53-bbe8-31fc85b64434/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c48f4fb176510aab4e75794b67c44712c8176202b96ce6e3ecc50b6bea99a0d/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69/volumes/kubernetes.io~projected/kube-api-access-trqwx DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80b9b535-bd12-4e37-8b22-bf77f7cd93db/volumes/kubernetes.io~projected/kube-api-access-jzs2k DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:442 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c11852f-4b27-4820-bfae-f65e13a3ceaa/volumes/kubernetes.io~projected/kube-api-access-f7pkr DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6df5824-c856-4945-b1da-7027bf549f00/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:377 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ecaf655a5749f31c47ffeef482f9d7254bf18093c12b077860d4e983bbc0e5a/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ecaf655a5749f3 MacAddress:0a:ec:70:0f:76:9f Speed:10000 Mtu:8901} {Name:31af91240135301 MacAddress:16:6f:e4:d0:7a:42 Speed:10000 Mtu:8901} {Name:3d501b0fe142e88 MacAddress:0e:d6:29:a7:74:68 Speed:10000 Mtu:8901} {Name:4acd952ae93fe5e MacAddress:96:ad:03:fa:db:29 Speed:10000 Mtu:8901} {Name:85257a07132006e MacAddress:06:a2:a0:82:65:78 Speed:10000 Mtu:8901} {Name:8869937adcf842e MacAddress:52:19:f8:65:28:a2 Speed:10000 Mtu:8901} {Name:9ab081a13476dd2 MacAddress:0e:02:03:68:6e:f5 Speed:10000 Mtu:8901} {Name:a6e4ebc826a0f45 MacAddress:02:49:de:1d:64:86 Speed:10000 Mtu:8901} {Name:b025b25eea78f03 MacAddress:9a:37:f1:f0:7e:4e Speed:10000 Mtu:8901} {Name:b6c14f265149f7e MacAddress:8e:a5:f1:ea:5a:d5 Speed:10000 Mtu:8901} {Name:b86fef7e87926cb MacAddress:12:64:f5:1d:d1:96 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:df:5e:c8:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:06:ea:74:5e:ef Speed:0 Mtu:8901} {Name:c4b463ce8c2b283 MacAddress:f2:ab:2d:cb:e3:29 Speed:10000 Mtu:8901} {Name:d52d8b31c3970d7 MacAddress:b2:05:53:1a:2e:ab Speed:10000 Mtu:8901} {Name:e04527b70270176 MacAddress:5e:9b:34:e2:b6:08 Speed:10000 Mtu:8901} {Name:e0d4acf00d9ad7e MacAddress:66:06:29:e7:20:e6 Speed:10000 Mtu:8901} {Name:ed9041b12c94810 MacAddress:32:76:ba:09:41:83 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:df:5e:c8:c1 Speed:0 Mtu:9001} {Name:f9f5321cbae3ae8 MacAddress:9e:39:78:94:ec:f0 Speed:10000 Mtu:8901} {Name:fbcab4d1329ae86 MacAddress:9e:95:a7:e7:a7:6a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:02:3c:73:db:8f:18 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:27:74:b7:fc:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.110954 6273 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111042 6273 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111395 6273 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111424 6273 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-103.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111567 6273 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111579 6273 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111594 6273 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111623 6273 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111755 6273 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.111956 6273 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112096 6273 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112133 6273 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112152 6273 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112165 6273 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112180 6273 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:17:26.112163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.112191 6273 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:17:26.114355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.113206 6273 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:17:26.114355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.113219 6273 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:17:26.114355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.113937 6273 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:17:26.117641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.117624 6273 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:17:26.117641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.117637 6273 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:17:26.117771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.117742 6273 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 06:17:26.117939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.117927 6273 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:17:26.118006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.117961 6273 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:17:26.118114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118099 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:17:26.118114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118116 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118121 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118128 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118139 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118148 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118157 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118166 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118178 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118188 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118199 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118215 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118247 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:17:26.118256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.118256 6273 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:17:26.122186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.122169 6273 server.go:1263] "Started kubelet" Nov 02 06:17:26.122362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.122273 6273 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:17:26.122802 ip-10-0-142-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:17:26.122937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.122888 6273 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:17:26.122937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.122880 6273 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:17:26.123213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.123195 6273 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:17:26.124818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.124108 6273 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:17:26.128588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.128568 6273 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.136057 6273 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.136107 6273 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.136137 6273 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.136196 6273 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.137133 6273 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.137187 6273 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:11:12 +0000 UTC" deadline="2028-04-01 15:56:37.563978393 +0000 UTC" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.137341 6273 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12393h39m11.426645788s" Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.137713 6273 factory.go:55] Registering systemd factory Nov 02 06:17:26.138863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.137735 6273 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.140173 6273 factory.go:153] Registering CRI-O factory Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.140206 6273 factory.go:223] Registration of the crio container factory successfully Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141014 6273 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141042 6273 factory.go:103] Registering Raw factory Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141060 6273 manager.go:1199] Started watching for new ooms in manager Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141122 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58618324-800f-4705-87d8-d68e27813002" volumeName="kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141171 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58618324-800f-4705-87d8-d68e27813002" volumeName="kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141185 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141198 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141211 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141222 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab2c88a-127f-4391-870a-f9757c5f9b04" volumeName="kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141235 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141247 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141261 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa211aa-bb54-4375-bb65-793ae2e67721" volumeName="kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141272 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141286 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe8c4326-8500-4a72-848f-7281b21b4853" volumeName="kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141298 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141310 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141322 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b70d77d-9aaa-44d6-8f1b-e68febbd921c" volumeName="kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141337 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe97868d-2a06-4ec7-a874-d88a2dfd0978" volumeName="kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 06:17:26.141347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141349 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe97868d-2a06-4ec7-a874-d88a2dfd0978" volumeName="kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141365 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141380 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5c9a49-a662-4b53-bbe8-31fc85b64434" volumeName="kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141395 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141408 6273 manager.go:319] Starting recovery of all containers Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.141408 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9ab1b6-621a-430c-9669-1071af47a675" volumeName="kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142547 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142588 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142604 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb289fe-8e1a-4abe-a37b-4cc6ac343150" volumeName="kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142618 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142632 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142646 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142661 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa211aa-bb54-4375-bb65-793ae2e67721" volumeName="kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142674 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142693 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142706 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" volumeName="kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142718 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142730 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2abb23a4-08db-4159-8715-df445adc7a71" volumeName="kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142742 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe8c4326-8500-4a72-848f-7281b21b4853" volumeName="kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142755 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142767 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd659865-edfc-4bb6-9c1c-38e02c8fe1e4" volumeName="kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142778 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb289fe-8e1a-4abe-a37b-4cc6ac343150" volumeName="kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142790 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142801 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa211aa-bb54-4375-bb65-793ae2e67721" volumeName="kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142813 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142824 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142834 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe97868d-2a06-4ec7-a874-d88a2dfd0978" volumeName="kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142845 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142857 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142868 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" volumeName="kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142881 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142892 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" volumeName="kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142904 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142929 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142941 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142952 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142963 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.142974 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72bc39c2-67ef-463a-ac5c-cc48dc9e00e8" volumeName="kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143001 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b9b535-bd12-4e37-8b22-bf77f7cd93db" volumeName="kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143014 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143024 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143034 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143054 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143065 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143077 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143089 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143102 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143115 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" volumeName="kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143128 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143142 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2abb23a4-08db-4159-8715-df445adc7a71" volumeName="kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143154 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5c9a49-a662-4b53-bbe8-31fc85b64434" volumeName="kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143167 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143179 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143191 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143205 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c11852f-4b27-4820-bfae-f65e13a3ceaa" volumeName="kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143219 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143233 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143245 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143257 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143270 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd659865-edfc-4bb6-9c1c-38e02c8fe1e4" volumeName="kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143282 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143296 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143313 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143325 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143338 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64c2f13d-6a20-4247-b926-ad21d58dc644" volumeName="kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143351 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143365 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5c9a49-a662-4b53-bbe8-31fc85b64434" volumeName="kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143377 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" volumeName="kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143391 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143405 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143419 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143432 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143445 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5c9a49-a662-4b53-bbe8-31fc85b64434" volumeName="kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143459 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69" volumeName="kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143471 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143484 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143496 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe8c4326-8500-4a72-848f-7281b21b4853" volumeName="kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143508 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143522 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143534 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143548 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:26.143509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143561 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853c435f-0786-450f-bdd9-f95cc551ba78" volumeName="kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143574 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd659865-edfc-4bb6-9c1c-38e02c8fe1e4" volumeName="kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143587 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143600 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c11852f-4b27-4820-bfae-f65e13a3ceaa" volumeName="kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143613 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" volumeName="kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143626 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58618324-800f-4705-87d8-d68e27813002" volumeName="kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143641 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853c435f-0786-450f-bdd9-f95cc551ba78" volumeName="kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143653 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143666 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143681 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143694 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143708 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b9b535-bd12-4e37-8b22-bf77f7cd93db" volumeName="kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143721 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143733 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa211aa-bb54-4375-bb65-793ae2e67721" volumeName="kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143746 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe97868d-2a06-4ec7-a874-d88a2dfd0978" volumeName="kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143760 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143772 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143803 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143816 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" volumeName="kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143828 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143842 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37012b0f-1510-431f-9811-f97e41935c40" volumeName="kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143855 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143870 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143883 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143896 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6df5824-c856-4945-b1da-7027bf549f00" volumeName="kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143907 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa211aa-bb54-4375-bb65-793ae2e67721" volumeName="kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143919 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" volumeName="kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143933 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143945 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58618324-800f-4705-87d8-d68e27813002" volumeName="kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143959 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b424524e-6361-4ccb-9de6-58c78d4021fa" volumeName="kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.143972 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae6eb4c-062d-4118-9bc0-8ac9af0cc881" volumeName="kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144001 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37012b0f-1510-431f-9811-f97e41935c40" volumeName="kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144015 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="206c88d3-cdff-496a-ac41-e8e7feca0ac8" volumeName="kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144028 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b70d77d-9aaa-44d6-8f1b-e68febbd921c" volumeName="kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144041 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c9168b-aa81-46fc-a51a-ec217aa704df" volumeName="kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144052 6273 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" volumeName="kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144064 6273 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:17:26.147387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.144071 6273 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:17:26.170095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.170065 6273 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:17:26.171542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.171511 6273 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:17:26.171542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.171531 6273 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:17:26.171688 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.171552 6273 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:17:26.171688 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.171585 6273 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 06:17:26.199938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.199873 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222275 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="c56d319523a90db1f3a399260ef6751e53b7fbcc70747373b876292e3feb97e3" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222304 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="a6d7a54875d56724e3917f5cc353e9bfe95f5be57dbf7450568cc84d27155c00" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222313 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="566ef63d9f18245b8b5ac28fafc091aa718772ecf5c875b01db44127b99654c6" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222321 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="6a00855691fd0f45884c4425b013e425f93b29504aa624d7ae31a46e573f1b1b" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222328 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="911f319e5473976af60f41f128bffd7990a3caac8b6f81154b641432b7fb3bc8" exitCode=0 Nov 02 06:17:26.222540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.222336 6273 generic.go:356] "Generic (PLEG): container finished" podID="fd25d9ad-68e0-49c8-a9cf-0ca18a976680" containerID="2229b3416aafeeeebedd2e6b315b31746e76f82efcf16eca6503f77eceb7b630" exitCode=0 Nov 02 06:17:26.232501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.232210 6273 generic.go:356] "Generic (PLEG): container finished" podID="20ebe6ec-1b67-42bd-8f17-873ea5c799a4" containerID="4bc0c5add56952e7cd1836c55c802ca5648a296f56fa093bb908299d3aeddb48" exitCode=0 Nov 02 06:17:26.236213 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.236129 6273 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" Nov 02 06:17:26.271686 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.271659 6273 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 06:17:26.271871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.271841 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715" exitCode=0 Nov 02 06:17:26.307750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.307721 6273 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:26.309210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.309186 6273 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:26.309394 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.309377 6273 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 06:17:26.318763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.318733 6273 generic.go:356] "Generic (PLEG): container finished" podID="bd6c06b094cb0b06c560a952f3f0bc9d" containerID="6a248642ba3fd57a4f6beb75efd8114cd59d8437e84dc0f34f011acd10d07cab" exitCode=0 Nov 02 06:17:26.328014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.327983 6273 manager.go:324] Recovery completed Nov 02 06:17:26.336212 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.336192 6273 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-103.us-west-2.compute.internal\" not found" Nov 02 06:17:26.353431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.353403 6273 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:26.358750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358730 6273 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:17:26.358750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358747 6273 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:17:26.358878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358769 6273 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:17:26.358925 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358899 6273 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 06:17:26.358925 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358910 6273 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 06:17:26.359016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358927 6273 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 06:17:26.359016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358935 6273 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 06:17:26.359016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358943 6273 policy_none.go:50] "Start" Nov 02 06:17:26.359016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358961 6273 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:17:26.359016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.358976 6273 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:17:26.359201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.359138 6273 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 06:17:26.359201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.359149 6273 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 06:17:26.359201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.359157 6273 policy_none.go:44] "Start" Nov 02 06:17:26.366315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.366282 6273 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:17:26.369392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.369374 6273 manager.go:341] "Starting Device Plugin manager" Nov 02 06:17:26.370090 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.369421 6273 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:17:26.370090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370032 6273 server.go:87] "Starting device plugin registration server" Nov 02 06:17:26.370350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370332 6273 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:17:26.370446 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370350 6273 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:17:26.370505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370471 6273 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:17:26.370644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370580 6273 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:17:26.370708 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370654 6273 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:17:26.370708 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370662 6273 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:17:26.370982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.370754 6273 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:17:26.370642525Z","UUID":"57c6c69e-0884-4b62-9b23-4878c207dfd7","Handler":null,"Name":"","Endpoint":""} Nov 02 06:17:26.372371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.372350 6273 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:17:26.372601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.372388 6273 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:17:26.373868 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.373760 6273 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:17:26.401198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.401170 6273 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:26.471258 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.471188 6273 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:17:26.472570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.472521 6273 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal"] Nov 02 06:17:26.473260 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473244 6273 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.473401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473351 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"05718cfa86289f66de168b803f5f9362","Type":"ContainerStarted","Data":"af6327a04ee741b95aa057554735df7633bf3d9c7ce9ff15e991ea1636fd8396"} Nov 02 06:17:26.473555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473409 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"05718cfa86289f66de168b803f5f9362","Type":"ContainerStarted","Data":"45407c5729a220be06b573bafb46edb217401971ef8dbf93d82ce7548a020f53"} Nov 02 06:17:26.473555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473517 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerStarted","Data":"a96fb4de76c154622e10bfc1daedfef5c57b1eb43c539c541d6d7d248fc19a6c"} Nov 02 06:17:26.473555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473541 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerDied","Data":"6a248642ba3fd57a4f6beb75efd8114cd59d8437e84dc0f34f011acd10d07cab"} Nov 02 06:17:26.473727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473560 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" event={"ID":"bd6c06b094cb0b06c560a952f3f0bc9d","Type":"ContainerStarted","Data":"18dd850d13f45eff6aa1d98ea9fb55c8b802cbf6cb6c9f698177fe67b9be9a73"} Nov 02 06:17:26.473727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.473568 6273 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.474611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.474592 6273 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:17:26.474686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.474624 6273 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:17:26.474686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.474639 6273 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:17:26.474794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.474691 6273 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.532512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.532483 6273 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:26.532613 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.532544 6273 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.533134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.533116 6273 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:26.533211 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:26.533152 6273 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.580890 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.580865 6273 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.581009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.580928 6273 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.610723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.610694 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.610824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.610729 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.610824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.610757 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711297 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711269 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711297 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711303 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711324 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711369 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05718cfa86289f66de168b803f5f9362-config\") pod \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"05718cfa86289f66de168b803f5f9362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711379 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:26.711499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:26.711377 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd6c06b094cb0b06c560a952f3f0bc9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" (UID: \"bd6c06b094cb0b06c560a952f3f0bc9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:27.114353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.114319 6273 apiserver.go:52] "Watching apiserver" Nov 02 06:17:27.258729 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.258700 6273 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:17:27.259683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.259646 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64f9df6957-68857","kube-system/global-pull-secret-syncer-h5h5g","openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc","openshift-console/console-6c9f964984-9rxlg","openshift-ingress-canary/ingress-canary-cqq7v","openshift-dns/dns-default-htss2","openshift-monitoring/prometheus-operator-9cd89b74c-shhdr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb","openshift-multus/network-metrics-daemon-s4zwp","openshift-monitoring/monitoring-plugin-77f4f76594-skv9z","openshift-network-operator/iptables-alerter-7lsf6","openshift-image-registry/node-ca-v9gfs","openshift-ovn-kubernetes/ovnkube-node-bzgdv","openshift-cluster-node-tuning-operator/tuned-gtvjg","openshift-console/console-58f78595c7-79l6n","openshift-multus/multus-29z9j","openshift-multus/multus-additional-cni-plugins-88wtj","openshift-dns/node-resolver-kcl2z","openshift-insights/insights-runtime-extractor-hmlst","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-75d975db85-8dlws","openshift-network-diagnostics/network-check-target-vd4bl","kube-system/konnectivity-agent-gqnpb","kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal","openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml","openshift-console/downloads-6dd789c8b7-kkplp","openshift-monitoring/node-exporter-7xg9n"] Nov 02 06:17:27.310327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310296 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:17:27.310327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310316 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.310537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310434 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.310537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310445 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:17:27.310537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310449 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.310537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310430 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:17:27.310537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310507 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310546 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310553 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310546 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310604 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310657 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:17:27.310795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310733 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:17:27.311113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.310853 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:17:27.311265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.311240 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:17:27.312216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.312196 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:17:27.312675 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.312659 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.323855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.323836 6273 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:27.323968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.323931 6273 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:27.340343 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.340324 6273 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:17:27.347544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.347522 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:17:27.347664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.347647 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:17:27.348303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348271 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.348405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348337 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.348471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348436 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:17:27.348527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348490 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.348634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348614 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:17:27.348741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348690 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:17:27.348842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.348821 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.349036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.349021 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:17:27.349163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.349149 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:17:27.349372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.349356 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.349591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.349570 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:17:27.351821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.351720 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.351821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.351741 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:17:27.351821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.351773 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.352439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352147 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:17:27.352439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352192 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:17:27.352662 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352640 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.352755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352722 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.352870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352854 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:17:27.352926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352909 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:17:27.353313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.352976 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:17:27.353855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.353781 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:17:27.354345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.354256 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:17:27.354567 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.354454 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:17:27.354951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.354933 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 06:17:27.355413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.354933 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.355538 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.355476 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.355726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.355599 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 06:17:27.355726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.355632 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 06:17:27.355726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.355635 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 06:17:27.355726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.355687 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 06:17:27.356266 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356248 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:17:27.356337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356273 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.356451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356336 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:17:27.356451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356412 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:17:27.356451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356423 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.356451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356252 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:17:27.356716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356489 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:17:27.356716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356544 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.356716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356608 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.356716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356544 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:17:27.356716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.356546 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:17:27.357104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357083 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 06:17:27.357231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357209 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l2vqf\"" Nov 02 06:17:27.357551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357364 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 06:17:27.357551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357455 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbcc6\"" Nov 02 06:17:27.357551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357456 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357585 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357665 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357672 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357695 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357692 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357712 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 06:17:27.357785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357733 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357797 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357802 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357841 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357846 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357880 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357889 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357896 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vdcff\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357679 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357843 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357945 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2dzzq\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357960 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358000 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9sdpq\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.357922 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358032 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358052 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358078 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358099 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 06:17:27.358148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358081 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358169 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358197 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358227 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358284 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358322 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358411 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358497 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358513 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358621 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fq2h9\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358620 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358750 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qnwd4\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358833 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5mfb9\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358855 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8hjs78d5780tl\"" Nov 02 06:17:27.358942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358914 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.358973 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359011 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ddeqh841j2mi8\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359070 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359266 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359339 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359372 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359414 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 06:17:27.359491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359467 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 06:17:27.359826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359530 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 06:17:27.359826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359560 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 06:17:27.359826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.359611 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tx8jf\"" Nov 02 06:17:27.393932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.393885 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 06:17:27.393932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.393912 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cmgkj\"" Nov 02 06:17:27.394102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394089 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 06:17:27.394160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394091 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8h274\"" Nov 02 06:17:27.394213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394172 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 06:17:27.394213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394202 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d58hj\"" Nov 02 06:17:27.394319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394275 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-144i712719v8e\"" Nov 02 06:17:27.394569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.394553 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 06:17:27.404522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.404465 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 06:17:27.404619 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.404592 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 06:17:27.404815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.404780 6273 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:27.404947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.404843 6273 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:27.405070 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:27.404984 6273 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:27.405139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.404884 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 06:17:27.405139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.405100 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 06:17:27.405248 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:27.405190 6273 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal" Nov 02 06:17:27.405248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.405198 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 06:17:27.415574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415549 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.415687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415589 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.415687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415619 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.415687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415647 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415690 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415724 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415751 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415777 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415799 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.415826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415823 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415845 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415870 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415890 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415903 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-iptables-alerter-script\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415921 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415947 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.415975 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416022 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416043 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416051 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-tuned\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416064 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416088 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416089 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-metrics-server-audit-profiles\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416110 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtbc\" (UniqueName: \"kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416127 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416153 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416175 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416185 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416284 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58618324-800f-4705-87d8-d68e27813002-metrics-tls\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416371 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416432 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416559 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416565 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416669 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.416775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416759 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416853 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-metrics-client-ca\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.416978 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-metrics-certs\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417027 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2abb23a4-08db-4159-8715-df445adc7a71-serviceca\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417053 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417221 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417242 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c11852f-4b27-4820-bfae-f65e13a3ceaa-cert\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417244 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclbn\" (UniqueName: \"kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417280 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.417333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417321 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417370 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417443 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417507 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417531 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b424524e-6361-4ccb-9de6-58c78d4021fa-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417572 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417609 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417636 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417674 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417689 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417742 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417775 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.417779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417776 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417796 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417805 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417851 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417857 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b9b535-bd12-4e37-8b22-bf77f7cd93db-tmp\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417888 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417909 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417954 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.417978 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418015 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418038 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418075 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418086 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418093 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418112 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418140 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418157 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzs2k\" (UniqueName: \"kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418178 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418203 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418267 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fx8\" (UniqueName: \"kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418267 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418284 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.418303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418309 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418337 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418361 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418398 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418413 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c9a49-a662-4b53-bbe8-31fc85b64434-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418415 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418444 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418459 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418461 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418487 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418557 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418583 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418612 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418651 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418676 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418699 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418722 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418742 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f7f\" (UniqueName: \"kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418770 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418793 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418811 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418836 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418861 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418883 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fzk\" (UniqueName: \"kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418901 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418922 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418961 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.418977 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419027 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419209 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xmh\" (UniqueName: \"kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419240 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419266 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419282 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.419299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419300 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419326 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419347 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419373 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419399 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419426 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419445 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419461 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419485 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419510 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419545 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419565 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419589 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419607 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419760 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419792 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419895 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-env-overrides\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419962 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.419615 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-serving-certs-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420133 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420164 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420179 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420193 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420213 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420235 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420251 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420274 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420299 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420336 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-binary-copy\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420336 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420381 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420556 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqwx\" (UniqueName: \"kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:27.420570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420586 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420615 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420629 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420642 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420713 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-audit-log\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420740 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv2m\" (UniqueName: \"kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420767 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420796 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fg8\" (UniqueName: \"kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420833 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420864 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420893 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420920 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420947 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420952 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.420975 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p26\" (UniqueName: \"kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421020 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421049 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421076 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421103 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421130 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblwh\" (UniqueName: \"kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421152 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421176 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421195 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421215 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421237 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421262 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421290 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421313 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421341 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421344 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421368 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421387 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421395 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-tmp\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421404 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421420 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421438 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421455 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421483 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421512 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421533 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421564 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/daa211aa-bb54-4375-bb65-793ae2e67721-ovn-node-metrics-cert\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421579 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pkr\" (UniqueName: \"kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421607 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421637 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421641 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58618324-800f-4705-87d8-d68e27813002-tmp-dir\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421667 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421173 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-tls\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.421964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.421833 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422032 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422080 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422229 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-script-lib\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422271 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422370 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/daa211aa-bb54-4375-bb65-793ae2e67721-ovnkube-config\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422430 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422616 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422723 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.422823 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-tls\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423060 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423218 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/206c88d3-cdff-496a-ac41-e8e7feca0ac8-metrics-client-ca\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423240 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-client-ca-bundle\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423387 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-cni-binary-copy\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423447 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-grpc-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.423822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423665 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.423932 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424056 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424128 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-secret-metrics-server-client-certs\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424221 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424313 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424374 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.424411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424408 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe97868d-2a06-4ec7-a874-d88a2dfd0978-data-volume\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.424727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424561 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.424786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.424769 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.425225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425205 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.425324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425223 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.425324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425276 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-textfile\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.425446 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425330 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.425446 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425391 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.425544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425453 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fab2c88a-127f-4391-870a-f9757c5f9b04-original-pull-secret\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.425544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425499 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.425654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425578 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.425654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425614 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.425786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425768 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe8c4326-8500-4a72-848f-7281b21b4853-multus-daemon-config\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.425786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425769 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-federate-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.425948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425871 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-metrics-client-ca\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.426029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.425953 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.426083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426032 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/206c88d3-cdff-496a-ac41-e8e7feca0ac8-tmp\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.426129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426080 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.426129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426089 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.426209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426178 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.426292 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426250 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.426328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426299 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.426418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426405 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.426458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426440 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.426505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426490 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.426540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426526 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.426540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426533 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.426597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426494 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba5c9a49-a662-4b53-bbe8-31fc85b64434-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.426683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426667 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.426739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426722 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.426786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426734 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkbd\" (UniqueName: \"kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.426786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426778 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.426904 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.426884 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.427050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427030 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.427114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427078 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.427164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427134 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlv2\" (UniqueName: \"kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.427212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427170 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.427278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427265 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.427319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427294 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwx9\" (UniqueName: \"kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.427501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427486 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.427562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427527 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkxk\" (UniqueName: \"kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.427562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427537 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-tls\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.427648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427574 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.427701 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427667 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.427833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427788 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.427912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427851 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.427912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427899 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.428037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.427959 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.428037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428016 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37012b0f-1510-431f-9811-f97e41935c40-tmp-dir\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.428157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428022 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.428157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428092 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpxt\" (UniqueName: \"kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.428157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428091 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.428157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428130 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.428344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428160 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.428344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428175 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58618324-800f-4705-87d8-d68e27813002-config-volume\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.428344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428220 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.428344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428234 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-6361-4ccb-9de6-58c78d4021fa-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.428519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428345 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:27.428519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428473 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe97868d-2a06-4ec7-a874-d88a2dfd0978-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.428519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428472 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.428678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428535 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.428678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428552 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.428678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428599 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.428678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428650 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:17:27.428678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428663 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:27.428892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428698 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.428892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428734 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/206c88d3-cdff-496a-ac41-e8e7feca0ac8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.428892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428787 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.428892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428883 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgkr\" (UniqueName: \"kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428914 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.428949 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429017 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429051 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429054 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853c435f-0786-450f-bdd9-f95cc551ba78-konnectivity-ca\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:17:27.429082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429075 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqbc\" (UniqueName: \"kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:27.429365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429103 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7bxsc\" (UID: \"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:17:27.429365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429128 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.429365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429154 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.429365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429280 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853c435f-0786-450f-bdd9-f95cc551ba78-agent-certs\") pod \"konnectivity-agent-gqnpb\" (UID: \"853c435f-0786-450f-bdd9-f95cc551ba78\") " pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:17:27.429365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429349 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6df5824-c856-4945-b1da-7027bf549f00-telemeter-client-tls\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.429569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429430 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.429569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429461 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72bc39c2-67ef-463a-ac5c-cc48dc9e00e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7bxsc\" (UID: \"72bc39c2-67ef-463a-ac5c-cc48dc9e00e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:17:27.429569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.429489 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.468654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.468627 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtbc\" (UniqueName: \"kubernetes.io/projected/ff9ab1b6-621a-430c-9669-1071af47a675-kube-api-access-chtbc\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.468773 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.468754 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclbn\" (UniqueName: \"kubernetes.io/projected/d6df5824-c856-4945-b1da-7027bf549f00-kube-api-access-tclbn\") pod \"telemeter-client-75d975db85-8dlws\" (UID: \"d6df5824-c856-4945-b1da-7027bf549f00\") " pod="openshift-monitoring/telemeter-client-75d975db85-8dlws" Nov 02 06:17:27.468846 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.468829 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"prometheus-k8s-0\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:27.471338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.471317 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"console-6c9f964984-9rxlg\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:27.473563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.473540 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fg8\" (UniqueName: \"kubernetes.io/projected/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-kube-api-access-d2fg8\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.473836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.473808 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fx8\" (UniqueName: \"kubernetes.io/projected/fe97868d-2a06-4ec7-a874-d88a2dfd0978-kube-api-access-j6fx8\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.473836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.473831 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzs2k\" (UniqueName: \"kubernetes.io/projected/80b9b535-bd12-4e37-8b22-bf77f7cd93db-kube-api-access-jzs2k\") pod \"downloads-6dd789c8b7-kkplp\" (UID: \"80b9b535-bd12-4e37-8b22-bf77f7cd93db\") " pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:27.474026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.473981 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p26\" (UniqueName: \"kubernetes.io/projected/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-kube-api-access-n6p26\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.475437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.475413 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv2m\" (UniqueName: \"kubernetes.io/projected/37012b0f-1510-431f-9811-f97e41935c40-kube-api-access-jzv2m\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.475723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.475696 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblwh\" (UniqueName: \"kubernetes.io/projected/ba5c9a49-a662-4b53-bbe8-31fc85b64434-kube-api-access-dblwh\") pod \"prometheus-operator-9cd89b74c-shhdr\" (UID: \"ba5c9a49-a662-4b53-bbe8-31fc85b64434\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-shhdr" Nov 02 06:17:27.475723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.475713 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqwx\" (UniqueName: \"kubernetes.io/projected/e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69-kube-api-access-trqwx\") pod \"network-metrics-daemon-s4zwp\" (UID: \"e68ba8ca-d3fa-43e1-b55c-f6217f8a5a69\") " pod="openshift-multus/network-metrics-daemon-s4zwp" Nov 02 06:17:27.475842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.475756 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f7f\" (UniqueName: \"kubernetes.io/projected/2ae6eb4c-062d-4118-9bc0-8ac9af0cc881-kube-api-access-b2f7f\") pod \"metrics-server-64f9df6957-68857\" (UID: \"2ae6eb4c-062d-4118-9bc0-8ac9af0cc881\") " pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:27.476589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.476547 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fzk\" (UniqueName: \"kubernetes.io/projected/b424524e-6361-4ccb-9de6-58c78d4021fa-kube-api-access-h7fzk\") pod \"kube-state-metrics-5dc8d7d89b-7c2lz\" (UID: \"b424524e-6361-4ccb-9de6-58c78d4021fa\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7c2lz" Nov 02 06:17:27.476859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.476838 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xmh\" (UniqueName: \"kubernetes.io/projected/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-kube-api-access-h2xmh\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.477303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.477281 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pkr\" (UniqueName: \"kubernetes.io/projected/2c11852f-4b27-4820-bfae-f65e13a3ceaa-kube-api-access-f7pkr\") pod \"ingress-canary-cqq7v\" (UID: \"2c11852f-4b27-4820-bfae-f65e13a3ceaa\") " pod="openshift-ingress-canary/ingress-canary-cqq7v" Nov 02 06:17:27.478262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.478243 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwx9\" (UniqueName: \"kubernetes.io/projected/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-kube-api-access-rbwx9\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.479491 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlv2\" (UniqueName: \"kubernetes.io/projected/2abb23a4-08db-4159-8715-df445adc7a71-kube-api-access-zdlv2\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.479737 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkxk\" (UniqueName: \"kubernetes.io/projected/58618324-800f-4705-87d8-d68e27813002-kube-api-access-qnkxk\") pod \"dns-default-htss2\" (UID: \"58618324-800f-4705-87d8-d68e27813002\") " pod="openshift-dns/dns-default-htss2" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.479803 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"console-58f78595c7-79l6n\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.479852 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkbd\" (UniqueName: \"kubernetes.io/projected/daa211aa-bb54-4375-bb65-793ae2e67721-kube-api-access-rdkbd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.479871 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"alertmanager-main-0\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.480387 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqbc\" (UniqueName: \"kubernetes.io/projected/7b70d77d-9aaa-44d6-8f1b-e68febbd921c-kube-api-access-cdqbc\") pod \"monitoring-plugin-77f4f76594-skv9z\" (UID: \"7b70d77d-9aaa-44d6-8f1b-e68febbd921c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.482044 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvn6\" (UniqueName: \"kubernetes.io/projected/64c2f13d-6a20-4247-b926-ad21d58dc644-kube-api-access-brvn6\") pod \"network-check-target-vd4bl\" (UID: \"64c2f13d-6a20-4247-b926-ad21d58dc644\") " pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.482162 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgkr\" (UniqueName: \"kubernetes.io/projected/206c88d3-cdff-496a-ac41-e8e7feca0ac8-kube-api-access-9wgkr\") pod \"thanos-querier-c5b54f4c7-hk8ml\" (UID: \"206c88d3-cdff-496a-ac41-e8e7feca0ac8\") " pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:27.483392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.482507 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpxt\" (UniqueName: \"kubernetes.io/projected/fe8c4326-8500-4a72-848f-7281b21b4853-kube-api-access-5xpxt\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.529852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.529823 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.529852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.529855 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.529875 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.529961 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-kubelet\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530028 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-systemd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530065 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-bin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530091 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.530129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530130 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530148 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530168 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-device-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530172 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530216 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-systemd\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530232 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-slash\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530234 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530273 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-host\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530287 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530311 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530316 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-lib-modules\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530334 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530356 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530371 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-kubelet\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530379 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2abb23a4-08db-4159-8715-df445adc7a71-host\") pod \"node-ca-v9gfs\" (UID: \"2abb23a4-08db-4159-8715-df445adc7a71\") " pod="openshift-image-registry/node-ca-v9gfs" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530412 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.530433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530442 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530466 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530479 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530483 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-socket-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530491 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530410 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-var-lib-cni-multus\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530514 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-netns\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530515 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-sys-fs\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530534 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-var-lib-kubelet\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530559 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530637 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530639 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-hostroot\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530665 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530691 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530700 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-k8s-cni-cncf-io\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530709 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530730 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-bin\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530735 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530743 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530760 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-cni-netd\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530760 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530785 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530794 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530802 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37012b0f-1510-431f-9811-f97e41935c40-hosts-file\") pod \"node-resolver-kcl2z\" (UID: \"37012b0f-1510-431f-9811-f97e41935c40\") " pod="openshift-dns/node-resolver-kcl2z" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530813 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530839 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530862 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530887 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530905 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-node-log\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530919 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530937 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530945 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-etc-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530954 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.530969 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531000 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531013 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-system-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531025 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531046 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-etc-kubernetes\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531044 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531081 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531086 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531104 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531123 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531147 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531167 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.531161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531190 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531208 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysctl-conf\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531212 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531244 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-netns\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531246 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531273 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb289fe-8e1a-4abe-a37b-4cc6ac343150-host-slash\") pod \"iptables-alerter-7lsf6\" (UID: \"0eb289fe-8e1a-4abe-a37b-4cc6ac343150\") " pod="openshift-network-operator/iptables-alerter-7lsf6" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531283 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531310 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531308 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-systemd-units\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531337 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531347 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-host-run-multus-certs\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531351 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531376 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-run\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531382 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531409 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-root\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531415 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531442 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531449 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-system-cni-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531465 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531473 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-var-lib-openvswitch\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531510 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-kubernetes\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531531 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531550 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531565 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531591 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531615 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531595 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-cni-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531640 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531668 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-conf-dir\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531713 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-multus-socket-dir-parent\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531718 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531749 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-run-ovn\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531756 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531804 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-kubelet-config\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531805 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-etc-selinux\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531836 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe97868d-2a06-4ec7-a874-d88a2dfd0978-crio-socket\") pod \"insights-runtime-extractor-hmlst\" (UID: \"fe97868d-2a06-4ec7-a874-d88a2dfd0978\") " pod="openshift-insights/insights-runtime-extractor-hmlst" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531844 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-sys\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531871 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/daa211aa-bb54-4375-bb65-793ae2e67721-log-socket\") pod \"ovnkube-node-bzgdv\" (UID: \"daa211aa-bb54-4375-bb65-793ae2e67721\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531872 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd25d9ad-68e0-49c8-a9cf-0ca18a976680-cnibin\") pod \"multus-additional-cni-plugins-88wtj\" (UID: \"fd25d9ad-68e0-49c8-a9cf-0ca18a976680\") " pod="openshift-multus/multus-additional-cni-plugins-88wtj" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531918 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-sysconfig\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531921 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-node-exporter-wtmp\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531549 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd659865-edfc-4bb6-9c1c-38e02c8fe1e4-etc-modprobe-d\") pod \"tuned-gtvjg\" (UID: \"cd659865-edfc-4bb6-9c1c-38e02c8fe1e4\") " pod="openshift-cluster-node-tuning-operator/tuned-gtvjg" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531950 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ebe6ec-1b67-42bd-8f17-873ea5c799a4-sys\") pod \"node-exporter-7xg9n\" (UID: \"20ebe6ec-1b67-42bd-8f17-873ea5c799a4\") " pod="openshift-monitoring/node-exporter-7xg9n" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.531966 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff9ab1b6-621a-430c-9669-1071af47a675-registration-dir\") pod \"aws-ebs-csi-driver-node-988qb\" (UID: \"ff9ab1b6-621a-430c-9669-1071af47a675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-988qb" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.532009 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fab2c88a-127f-4391-870a-f9757c5f9b04-dbus\") pod \"global-pull-secret-syncer-h5h5g\" (UID: \"fab2c88a-127f-4391-870a-f9757c5f9b04\") " pod="kube-system/global-pull-secret-syncer-h5h5g" Nov 02 06:17:27.532858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.532022 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe8c4326-8500-4a72-848f-7281b21b4853-cnibin\") pod \"multus-29z9j\" (UID: \"fe8c4326-8500-4a72-848f-7281b21b4853\") " pod="openshift-multus/multus-29z9j" Nov 02 06:17:27.839901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:27.839871 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:29.964011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:29.963966 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-htss2" Nov 02 06:17:30.816197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:30.816158 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:30.882361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:30.882333 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:30.917011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:30.916973 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:30.917921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:30.917900 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kkplp" Nov 02 06:17:31.839087 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:31.839059 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7bxsc" Nov 02 06:17:32.130630 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:32.130558 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c5b54f4c7-hk8ml" Nov 02 06:17:33.414958 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:33.414881 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:33.582285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:33.582252 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:17:33.738223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:33.738145 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:17:34.520042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:34.520010 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:34.776743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:34.776672 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:34.776911 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:34.776890 6273 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:17:35.683453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:35.683421 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-skv9z" Nov 02 06:17:36.755654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:36.755623 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64f9df6957-68857" Nov 02 06:17:38.552667 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:38.552630 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:17:39.295794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.295762 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69cdc8f4b-xs57d"] Nov 02 06:17:39.332486 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.332457 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.428263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.428235 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cdc8f4b-xs57d"] Nov 02 06:17:39.434849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.434829 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-oauth-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.434932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.434858 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-oauth-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.434932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.434887 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.435048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.435030 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.435090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.435078 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-tmp\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.435126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.435096 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-service-ca\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.435159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.435143 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-trusted-ca-bundle\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.435331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.435313 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrmz\" (UniqueName: \"kubernetes.io/projected/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-kube-api-access-fgrmz\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.535827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535798 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-tmp\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.535827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535832 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-service-ca\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535852 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-trusted-ca-bundle\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535934 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrmz\" (UniqueName: \"kubernetes.io/projected/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-kube-api-access-fgrmz\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535960 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-oauth-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.535982 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-oauth-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536039 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536210 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536247 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-tmp\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.536527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536499 6273 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:17:39.536838 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536812 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-service-ca\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.537006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536969 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-oauth-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.537092 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.536969 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.537256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.537234 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-trusted-ca-bundle\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.538767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.538748 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-oauth-config\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.538845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.538782 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-console-serving-cert\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.590791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.590709 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrmz\" (UniqueName: \"kubernetes.io/projected/b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c-kube-api-access-fgrmz\") pod \"console-69cdc8f4b-xs57d\" (UID: \"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c\") " pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:39.664777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:39.664747 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:40.060134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:40.060099 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cdc8f4b-xs57d"] Nov 02 06:17:40.068410 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:40.068373 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e0c0e0_5b8b_4225_a088_cd4a48e31e7c.slice/crio-74319b1a93458dbd01726e813a0abc951dd36b3606d0a77cdf736f50fe952035 WatchSource:0}: Error finding container 74319b1a93458dbd01726e813a0abc951dd36b3606d0a77cdf736f50fe952035: Status 404 returned error can't find the container with id 74319b1a93458dbd01726e813a0abc951dd36b3606d0a77cdf736f50fe952035 Nov 02 06:17:40.393114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:40.393029 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cdc8f4b-xs57d" event={"ID":"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c","Type":"ContainerStarted","Data":"dbbce390f7b32cc22f6cedee2ceff2596be480da9968fda2441f8d97b26b3435"} Nov 02 06:17:40.393114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:40.393089 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cdc8f4b-xs57d" event={"ID":"b9e0c0e0-5b8b-4225-a088-cd4a48e31e7c","Type":"ContainerStarted","Data":"74319b1a93458dbd01726e813a0abc951dd36b3606d0a77cdf736f50fe952035"} Nov 02 06:17:40.501094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:40.501022 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69cdc8f4b-xs57d" podStartSLOduration=1.50098114 podStartE2EDuration="1.50098114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:40.500288168 +0000 UTC m=+14.473622879" watchObservedRunningTime="2026-11-02 06:17:40.50098114 +0000 UTC m=+14.474315828" Nov 02 06:17:45.771736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.771700 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772002 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="alertmanager" containerID="cri-o://90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca" gracePeriod=120 Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772048 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-metric" containerID="cri-o://77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889" gracePeriod=120 Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772085 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-web" containerID="cri-o://fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d" gracePeriod=120 Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772110 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy" containerID="cri-o://98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212" gracePeriod=120 Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772135 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="prom-label-proxy" containerID="cri-o://f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2" gracePeriod=120 Nov 02 06:17:45.772271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:45.772155 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="config-reloader" containerID="cri-o://9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6" gracePeriod=120 Nov 02 06:17:46.419285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419243 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2" exitCode=0 Nov 02 06:17:46.419285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419271 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889" exitCode=0 Nov 02 06:17:46.419285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419280 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212" exitCode=0 Nov 02 06:17:46.419285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419288 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6" exitCode=0 Nov 02 06:17:46.419285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419295 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca" exitCode=0 Nov 02 06:17:46.419642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419311 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2"} Nov 02 06:17:46.419642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419372 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889"} Nov 02 06:17:46.419642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419397 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212"} Nov 02 06:17:46.419642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419420 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6"} Nov 02 06:17:46.419642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.419443 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca"} Nov 02 06:17:46.472871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:46.472841 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzgdv" Nov 02 06:17:47.073980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.073952 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.198695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198670 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.198859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198710 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.198859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198745 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.198859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198775 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.198859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198804 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.198859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198850 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.198923 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199021 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199053 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199247 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:47.199587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199527 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199675 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199849 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.199981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.199961 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:47.200166 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.200118 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\" (UID: \"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c\") " Nov 02 06:17:47.200362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.200337 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:47.200508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.200489 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-metrics-client-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.200587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.200516 6273 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-main-db\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.200587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.200535 6273 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.203843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.203744 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.204016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.203978 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.204274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.204253 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.204488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.204432 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.204488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.204264 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "kube-api-access-jhqxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:47.204615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.204563 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.204714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.204694 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:47.205505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.205480 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:47.208288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.208258 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.216151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.216125 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config" pod "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" (UID: "1d1e42d4-ad24-4090-bf7a-1af618a6ba1c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:47.301175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301138 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqxv\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-kube-api-access-jhqxv\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301167 6273 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-cluster-tls-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301178 6273 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301188 6273 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-web-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301197 6273 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-out\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301206 6273 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-tls-assets\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301214 6273 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301225 6273 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301234 6273 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-main-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.301404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.301243 6273 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:47.426335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.426304 6273 generic.go:356] "Generic (PLEG): container finished" podID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerID="fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d" exitCode=0 Nov 02 06:17:47.426481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.426371 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.426481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.426370 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d"} Nov 02 06:17:47.426481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.426470 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1d1e42d4-ad24-4090-bf7a-1af618a6ba1c","Type":"ContainerDied","Data":"a6e4ebc826a0f4533b4340f4e44b766f35b2e93f22aff90bb2c13e03be7b4d09"} Nov 02 06:17:47.426578 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.426510 6273 scope.go:122] "RemoveContainer" containerID="f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2" Nov 02 06:17:47.435135 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.435117 6273 scope.go:122] "RemoveContainer" containerID="77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889" Nov 02 06:17:47.442159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.442143 6273 scope.go:122] "RemoveContainer" containerID="98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212" Nov 02 06:17:47.448822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.448789 6273 scope.go:122] "RemoveContainer" containerID="fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d" Nov 02 06:17:47.455399 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.455382 6273 scope.go:122] "RemoveContainer" containerID="9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6" Nov 02 06:17:47.462770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.462752 6273 scope.go:122] "RemoveContainer" containerID="90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca" Nov 02 06:17:47.470177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.470148 6273 scope.go:122] "RemoveContainer" containerID="8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715" Nov 02 06:17:47.478442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.478421 6273 scope.go:122] "RemoveContainer" containerID="f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2" Nov 02 06:17:47.478756 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.478698 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2\": container with ID starting with f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2 not found: ID does not exist" containerID="f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2" Nov 02 06:17:47.478873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.478737 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2"} err="failed to get container status \"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2\": rpc error: code = NotFound desc = could not find container \"f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2\": container with ID starting with f8cd79e6b21cd27a18498477f5183cfd04eef08c6590e6526ab0d476ebe211a2 not found: ID does not exist" Nov 02 06:17:47.478873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.478768 6273 scope.go:122] "RemoveContainer" containerID="77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889" Nov 02 06:17:47.479039 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.479026 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889\": container with ID starting with 77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889 not found: ID does not exist" containerID="77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889" Nov 02 06:17:47.479077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479047 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889"} err="failed to get container status \"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889\": rpc error: code = NotFound desc = could not find container \"77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889\": container with ID starting with 77b232b8915f3e4a994e1163bab909b4e230e30f67642bebd421f784ee402889 not found: ID does not exist" Nov 02 06:17:47.479077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479061 6273 scope.go:122] "RemoveContainer" containerID="98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212" Nov 02 06:17:47.479353 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.479331 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212\": container with ID starting with 98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212 not found: ID does not exist" containerID="98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212" Nov 02 06:17:47.479395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479362 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212"} err="failed to get container status \"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212\": rpc error: code = NotFound desc = could not find container \"98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212\": container with ID starting with 98f84c7b5ef02c76aa8424cbfed6d4cbd94072776810eda23b22e6b830e37212 not found: ID does not exist" Nov 02 06:17:47.479395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479384 6273 scope.go:122] "RemoveContainer" containerID="fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d" Nov 02 06:17:47.479655 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.479638 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d\": container with ID starting with fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d not found: ID does not exist" containerID="fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d" Nov 02 06:17:47.479714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479659 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d"} err="failed to get container status \"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d\": rpc error: code = NotFound desc = could not find container \"fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d\": container with ID starting with fa044fa01d2c7058a565e93edc55e4eae7dce21efc9c73fb9878b9eb1ff5ab6d not found: ID does not exist" Nov 02 06:17:47.479714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479671 6273 scope.go:122] "RemoveContainer" containerID="9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6" Nov 02 06:17:47.479921 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.479900 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6\": container with ID starting with 9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6 not found: ID does not exist" containerID="9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6" Nov 02 06:17:47.479965 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479926 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6"} err="failed to get container status \"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6\": rpc error: code = NotFound desc = could not find container \"9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6\": container with ID starting with 9e5b61f67d8303da2a5c753d60a74677594792e999cd94531de43c491d4b77f6 not found: ID does not exist" Nov 02 06:17:47.479965 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.479941 6273 scope.go:122] "RemoveContainer" containerID="90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca" Nov 02 06:17:47.480181 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.480165 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca\": container with ID starting with 90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca not found: ID does not exist" containerID="90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca" Nov 02 06:17:47.480230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.480185 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca"} err="failed to get container status \"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca\": rpc error: code = NotFound desc = could not find container \"90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca\": container with ID starting with 90a8657f1e397d19ef51f99e28bc9608c6beb1a23fe52cfbbf33d687adbc53ca not found: ID does not exist" Nov 02 06:17:47.480230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.480199 6273 scope.go:122] "RemoveContainer" containerID="8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715" Nov 02 06:17:47.480418 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:47.480399 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715\": container with ID starting with 8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715 not found: ID does not exist" containerID="8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715" Nov 02 06:17:47.480505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.480420 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715"} err="failed to get container status \"8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715\": rpc error: code = NotFound desc = could not find container \"8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715\": container with ID starting with 8eafa53e5b879db5b636c5cc36dbf0d0280373cf68292fcc3b79ccf37e8ef715 not found: ID does not exist" Nov 02 06:17:47.583507 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.583472 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:47.587428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:47.587402 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:48.177547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:48.177516 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" path="/var/lib/kubelet/pods/1d1e42d4-ad24-4090-bf7a-1af618a6ba1c/volumes" Nov 02 06:17:49.665180 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:49.665144 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:49.665567 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:49.665208 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:49.670004 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:49.669966 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:50.441529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:50.441502 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69cdc8f4b-xs57d" Nov 02 06:17:50.596338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:50.596303 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:17:51.599595 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599560 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599855 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="prometheus" containerID="cri-o://d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" gracePeriod=600 Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599867 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" gracePeriod=600 Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599904 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-web" containerID="cri-o://a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" gracePeriod=600 Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599922 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="config-reloader" containerID="cri-o://693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" gracePeriod=600 Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.599899 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="thanos-sidecar" containerID="cri-o://1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" gracePeriod=600 Nov 02 06:17:51.600040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.600019 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy" containerID="cri-o://5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" gracePeriod=600 Nov 02 06:17:51.649785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.649751 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:51.900683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:51.900652 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.057645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057614 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.057819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057659 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.057819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057678 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.057819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057702 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.057980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057930 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.057976 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058025 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058061 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058088 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.058164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058110 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058146 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058181 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058226 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058264 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058299 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058334 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058372 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058402 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058450 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058482 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058513 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") pod \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\" (UID: \"f3b1ccb3-168c-43bd-8642-57e6e7b4f430\") " Nov 02 06:17:52.058892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058776 6273 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.058892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.058783 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:52.060755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.059263 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.060755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.059278 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.060755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.059578 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.060755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.060472 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.060755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.060523 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.061177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.061138 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:52.061939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.061912 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.062645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.062599 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:52.062775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.062653 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.062840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.062796 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "kube-api-access-9jx64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:52.062958 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.062901 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.063090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063037 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.063343 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063319 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.063466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063449 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.063524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063502 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.063755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063730 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:52.063829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.063735 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.074305 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.074284 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config" pod "f3b1ccb3-168c-43bd-8642-57e6e7b4f430" (UID: "f3b1ccb3-168c-43bd-8642-57e6e7b4f430"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:52.160075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160052 6273 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config-out\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160075 6273 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160086 6273 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160096 6273 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160105 6273 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-web-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160113 6273 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160122 6273 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160130 6273 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160138 6273 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-tls-assets\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160148 6273 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-db\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160156 6273 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160165 6273 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-kube-rbac-proxy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160174 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jx64\" (UniqueName: \"kubernetes.io/projected/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-kube-api-access-9jx64\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160182 6273 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160190 6273 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160198 6273 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-metrics-client-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160207 6273 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-configmap-metrics-client-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160215 6273 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.160220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.160223 6273 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3b1ccb3-168c-43bd-8642-57e6e7b4f430-secret-grpc-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448072 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" exitCode=0 Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448099 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" exitCode=0 Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448105 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" exitCode=0 Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448113 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" exitCode=0 Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448118 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" exitCode=0 Nov 02 06:17:52.448149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448122 6273 generic.go:356] "Generic (PLEG): container finished" podID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" exitCode=0 Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448147 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448167 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448242 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448269 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448294 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448309 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448318 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448341 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} Nov 02 06:17:52.448441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.448364 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3b1ccb3-168c-43bd-8642-57e6e7b4f430","Type":"ContainerDied","Data":"e04527b70270176d2190d6d0bf9322cfd80316e2026fcb5777cea2bce57a375c"} Nov 02 06:17:52.456072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.456054 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.463132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.463119 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.470002 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.469968 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.476692 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.476667 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.484167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.484133 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.493624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.493602 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.501166 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.501147 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.501489 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.501468 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.501544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.501499 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.501544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.501517 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.501806 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.501775 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.501849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.501817 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.501849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.501839 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.502169 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.502151 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.502218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502176 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.502218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502192 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.502435 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.502416 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.502479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502440 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.502479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502458 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.502677 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.502659 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.502726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502681 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.502726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502694 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.502888 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.502870 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.502935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502890 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.502935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.502902 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.504057 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:17:52.503173 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.504057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.503201 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.504057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.503220 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.505458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505422 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.505560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505459 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.505719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505699 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.505719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505718 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.505957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505940 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.506034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.505957 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.506229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506211 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.506335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506231 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.506477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506453 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.506515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506478 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.506697 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506676 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.506774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506698 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.506914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506899 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.506964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.506914 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.507179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507163 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.507179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507179 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.507417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507387 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.507466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507420 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.507696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507677 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.507743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507697 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.507926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507909 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.507975 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.507926 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.508164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508148 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.508221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508164 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.508401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508380 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.508464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508401 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.508637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508618 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.508686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508637 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.508881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508861 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.508933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.508880 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.509132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509115 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.509132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509131 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.509387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509371 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.509440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509395 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.509660 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509634 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.509717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509660 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.509920 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509899 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.509957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.509923 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.510198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510170 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.510198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510190 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.510445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510427 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.510513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510445 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.510668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510651 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.510746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510668 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.510893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510876 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.510959 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.510894 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.511162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511144 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.511213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511163 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.511370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511355 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.511416 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511370 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.511625 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511601 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.511625 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511625 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.511872 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511839 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.511923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.511873 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.512160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512139 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.512255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512160 6273 scope.go:122] "RemoveContainer" containerID="2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b" Nov 02 06:17:52.512463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512442 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b"} err="failed to get container status \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": rpc error: code = NotFound desc = could not find container \"2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b\": container with ID starting with 2743ee24f45616ba89f2ebbff3df9af436f690e61400c986ecb8fe9e9a913e9b not found: ID does not exist" Nov 02 06:17:52.512463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512462 6273 scope.go:122] "RemoveContainer" containerID="5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7" Nov 02 06:17:52.512754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512729 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7"} err="failed to get container status \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": rpc error: code = NotFound desc = could not find container \"5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7\": container with ID starting with 5c6cdaf9d02c31bcc208554444968b5e2f46d483007dedecd2b61d402381f5f7 not found: ID does not exist" Nov 02 06:17:52.512832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.512754 6273 scope.go:122] "RemoveContainer" containerID="a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b" Nov 02 06:17:52.513047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513025 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b"} err="failed to get container status \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": rpc error: code = NotFound desc = could not find container \"a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b\": container with ID starting with a434bf81516640da59c357ce55c4f6f43e4a01dc6284bee98d68a506589f783b not found: ID does not exist" Nov 02 06:17:52.513047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513046 6273 scope.go:122] "RemoveContainer" containerID="1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36" Nov 02 06:17:52.513315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513290 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36"} err="failed to get container status \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": rpc error: code = NotFound desc = could not find container \"1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36\": container with ID starting with 1679df4056e64724146e460651750d3695d823d10fca5673af235f7a4f84dc36 not found: ID does not exist" Nov 02 06:17:52.513402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513315 6273 scope.go:122] "RemoveContainer" containerID="693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19" Nov 02 06:17:52.513584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513562 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19"} err="failed to get container status \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": rpc error: code = NotFound desc = could not find container \"693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19\": container with ID starting with 693f5df7f421e6b5aa64d039f15f1f8c511c3cc7a14dfa26e181e5b7bcd5fd19 not found: ID does not exist" Nov 02 06:17:52.513584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513582 6273 scope.go:122] "RemoveContainer" containerID="d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb" Nov 02 06:17:52.513842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513815 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb"} err="failed to get container status \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": rpc error: code = NotFound desc = could not find container \"d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb\": container with ID starting with d7d8eca1608205e60dc46115ce6f8e1454a2057a862b9fc39c7249fba9feb3fb not found: ID does not exist" Nov 02 06:17:52.513919 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.513841 6273 scope.go:122] "RemoveContainer" containerID="9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de" Nov 02 06:17:52.514113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.514091 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de"} err="failed to get container status \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": rpc error: code = NotFound desc = could not find container \"9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de\": container with ID starting with 9f781b9a957709926e52f0e3921a757b49b4048a43364d09d72a036102eb83de not found: ID does not exist" Nov 02 06:17:52.604653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.604621 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:52.651639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.651511 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:52.652264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652246 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:52.652536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652525 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="init-config-reloader" Nov 02 06:17:52.652577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652537 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="init-config-reloader" Nov 02 06:17:52.652577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652549 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="alertmanager" Nov 02 06:17:52.652577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652554 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="alertmanager" Nov 02 06:17:52.652577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652564 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="prom-label-proxy" Nov 02 06:17:52.652577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652569 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="prom-label-proxy" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652580 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652584 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652591 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652595 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652601 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652605 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652612 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="init-config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652616 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="init-config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652624 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="prometheus" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652628 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="prometheus" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652638 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="thanos-sidecar" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652642 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="thanos-sidecar" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652654 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652658 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652665 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-metric" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652669 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-metric" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652675 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652679 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="config-reloader" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652685 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-thanos" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652690 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-thanos" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652697 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.652723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652701 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652749 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652756 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="config-reloader" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652761 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652767 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="init-config-reloader" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652773 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="prometheus" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652779 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="thanos-sidecar" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652785 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652790 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-metric" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652795 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="config-reloader" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652800 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="kube-rbac-proxy-thanos" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652804 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="kube-rbac-proxy-web" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652810 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" containerName="init-config-reloader" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652813 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="alertmanager" Nov 02 06:17:52.653339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.652818 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e42d4-ad24-4090-bf7a-1af618a6ba1c" containerName="prom-label-proxy" Nov 02 06:17:52.688970 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.688947 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.738851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.738775 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 06:17:52.739031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.738878 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 06:17:52.739031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.738910 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 06:17:52.739031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739027 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 06:17:52.739197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739076 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cmgkj\"" Nov 02 06:17:52.739197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739082 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 06:17:52.739282 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739200 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 06:17:52.739282 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739248 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 06:17:52.739373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739278 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 06:17:52.739373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739290 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 06:17:52.739373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739305 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 06:17:52.739373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739311 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 06:17:52.739373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739315 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ddeqh841j2mi8\"" Nov 02 06:17:52.739561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.739394 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 06:17:52.764145 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764120 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764151 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764183 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764333 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764369 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764411 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764475 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764514 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764544 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764618 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764643 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764671 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764700 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh7c\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-kube-api-access-phh7c\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764749 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764801 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.764837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764827 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.765164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.764863 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.765164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.765094 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.765164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.765127 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.765320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.765183 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.786651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.786628 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 06:17:52.786651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.786638 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 06:17:52.787503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.787484 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:52.865961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.865931 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.865965 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866005 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866028 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh7c\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-kube-api-access-phh7c\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866060 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866202 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866251 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866288 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866319 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866354 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866389 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866440 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866471 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866507 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866543 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866566 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866566 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866600 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.866627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866626 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866651 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866673 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866736 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.866813 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.867007 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.867110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.867070 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.869791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.867768 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.869791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.868045 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.870195 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.870585 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-web-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.870727 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.870820 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cbaba29-3141-44c5-ba74-b99b221a7762-config-out\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871091 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871274 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871472 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9cbaba29-3141-44c5-ba74-b99b221a7762-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871511 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871688 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-config\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871755 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.871933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.871917 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.872525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.872508 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9cbaba29-3141-44c5-ba74-b99b221a7762-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:52.920190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:52.920162 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh7c\" (UniqueName: \"kubernetes.io/projected/9cbaba29-3141-44c5-ba74-b99b221a7762-kube-api-access-phh7c\") pod \"prometheus-k8s-0\" (UID: \"9cbaba29-3141-44c5-ba74-b99b221a7762\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:53.130326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:53.130258 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gqnpb" Nov 02 06:17:53.137707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:53.137688 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:53.646367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:53.646336 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 06:17:53.654693 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:17:53.654663 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbaba29_3141_44c5_ba74_b99b221a7762.slice/crio-1b4007e59b714fa802d9d344b40d3eb9cc89084575671ceaf742b07a3291f726 WatchSource:0}: Error finding container 1b4007e59b714fa802d9d344b40d3eb9cc89084575671ceaf742b07a3291f726: Status 404 returned error can't find the container with id 1b4007e59b714fa802d9d344b40d3eb9cc89084575671ceaf742b07a3291f726 Nov 02 06:17:54.176725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:54.176694 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b1ccb3-168c-43bd-8642-57e6e7b4f430" path="/var/lib/kubelet/pods/f3b1ccb3-168c-43bd-8642-57e6e7b4f430/volumes" Nov 02 06:17:54.458515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:54.458449 6273 generic.go:356] "Generic (PLEG): container finished" podID="9cbaba29-3141-44c5-ba74-b99b221a7762" containerID="620f7e9d5b4372181e42f459ca069ea3dd5685171efb4815c2f93ca53b933356" exitCode=0 Nov 02 06:17:54.458515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:54.458482 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerDied","Data":"620f7e9d5b4372181e42f459ca069ea3dd5685171efb4815c2f93ca53b933356"} Nov 02 06:17:54.458681 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:54.458529 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"1b4007e59b714fa802d9d344b40d3eb9cc89084575671ceaf742b07a3291f726"} Nov 02 06:17:54.522652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:54.522630 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vd4bl" Nov 02 06:17:55.466738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.466700 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"e38f9d9ed5907077f22601385ae6140ff9643896fb8f1cdd241b8960c45e5b75"} Nov 02 06:17:55.467226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.466774 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"4bcce773f76227e118b0da394a73224f05cbd0dd18c8daf3715f23a23cb680dc"} Nov 02 06:17:55.467226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.466814 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"3834f5311aa73fcc7810975aeb3cde0b84e760d9c475d1a9044820bfa1b5748a"} Nov 02 06:17:55.467226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.466854 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"8c01c1df6f52789094f8a8b3f0ac941503016a54356969959f744da807672416"} Nov 02 06:17:55.467226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.467010 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"c8cb59bdb42128f87fb8e0fd34648d3c6704ae5d50dbfda4dafd77ba26fd57df"} Nov 02 06:17:55.467226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.467070 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9cbaba29-3141-44c5-ba74-b99b221a7762","Type":"ContainerStarted","Data":"d50f006a408b6ce9227e590aa31dd0f168383bb0ddd7fe6c830bc7332ef803c6"} Nov 02 06:17:55.584776 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:55.584726 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.572344348 podStartE2EDuration="3.584715359s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.371011ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:55.5838691 +0000 UTC m=+29.557203784" watchObservedRunningTime="2026-11-02 06:17:55.584715359 +0000 UTC m=+29.558050042" Nov 02 06:17:58.138837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:58.138800 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:17:59.387146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.387099 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c9f964984-9rxlg" podUID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" containerName="console" containerID="cri-o://87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de" gracePeriod=15 Nov 02 06:17:59.670395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.670372 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:17:59.823897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.823863 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.823903 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.823940 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824044 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824134 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824176 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824219 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") pod \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\" (UID: \"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52\") " Nov 02 06:17:59.824444 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824279 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:59.824444 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824331 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:59.824444 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824343 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:59.824587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824521 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:59.824647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824591 6273 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-oauth-serving-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.824647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824613 6273 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.824647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824631 6273 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.824746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.824647 6273 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-tmp\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.826400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.826372 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:59.826529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.826461 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "kube-api-access-h6q8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:59.826529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.826485 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config" pod "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" (UID: "2d2cc1bb-2724-4b77-80a6-20be4eb4ca52"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:59.925886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.925834 6273 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-oauth-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.925886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.925857 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6q8x\" (UniqueName: \"kubernetes.io/projected/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-kube-api-access-h6q8x\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:59.925886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:17:59.925869 6273 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52-console-serving-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:00.488234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.488149 6273 generic.go:356] "Generic (PLEG): container finished" podID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" containerID="87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de" exitCode=0 Nov 02 06:18:00.488234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.488196 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f964984-9rxlg" Nov 02 06:18:00.488654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.488236 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f964984-9rxlg" event={"ID":"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52","Type":"ContainerDied","Data":"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de"} Nov 02 06:18:00.488654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.488290 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f964984-9rxlg" event={"ID":"2d2cc1bb-2724-4b77-80a6-20be4eb4ca52","Type":"ContainerDied","Data":"8869937adcf842e332c61f12f77a91fe0a57e5d37348fb22850cbcbc19e578a7"} Nov 02 06:18:00.488654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.488312 6273 scope.go:122] "RemoveContainer" containerID="87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de" Nov 02 06:18:00.496125 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.496106 6273 scope.go:122] "RemoveContainer" containerID="87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de" Nov 02 06:18:00.496391 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:18:00.496375 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de\": container with ID starting with 87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de not found: ID does not exist" containerID="87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de" Nov 02 06:18:00.496440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.496400 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de"} err="failed to get container status \"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de\": rpc error: code = NotFound desc = could not find container \"87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de\": container with ID starting with 87f096e84030a3700f6d1f90ac3376f3d8db146f99611a88ddd4e7ce600580de not found: ID does not exist" Nov 02 06:18:00.642548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.642516 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:18:00.689178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:00.689145 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c9f964984-9rxlg"] Nov 02 06:18:02.176528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:02.176493 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" path="/var/lib/kubelet/pods/2d2cc1bb-2724-4b77-80a6-20be4eb4ca52/volumes" Nov 02 06:18:08.551099 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:18:08.551051 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:18:15.618453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.618389 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58f78595c7-79l6n" podUID="a7c9168b-aa81-46fc-a51a-ec217aa704df" containerName="console" containerID="cri-o://fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b" gracePeriod=15 Nov 02 06:18:15.870251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.870196 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:18:15.950619 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950592 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.950766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950624 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.950766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950645 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.950766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950689 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.950932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950835 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.951017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950941 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.951017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950947 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:18:15.951017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.950970 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.951172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951030 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") pod \"a7c9168b-aa81-46fc-a51a-ec217aa704df\" (UID: \"a7c9168b-aa81-46fc-a51a-ec217aa704df\") " Nov 02 06:18:15.951172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951145 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:18:15.951334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951201 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:18:15.951462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951363 6273 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-oauth-serving-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:15.951462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951383 6273 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7c9168b-aa81-46fc-a51a-ec217aa704df-tmp\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:15.951462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951385 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:18:15.951462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.951420 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:18:15.952968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.952947 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:18:15.953478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.953454 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:18:15.956031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:15.953609 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k" pod "a7c9168b-aa81-46fc-a51a-ec217aa704df" (UID: "a7c9168b-aa81-46fc-a51a-ec217aa704df"). InnerVolumeSpecName "kube-api-access-r7s8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:18:16.052635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052601 6273 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.052635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052634 6273 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-serving-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.052809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052651 6273 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-oauth-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.052809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052666 6273 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-trusted-ca-bundle\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.052809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052680 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s8k\" (UniqueName: \"kubernetes.io/projected/a7c9168b-aa81-46fc-a51a-ec217aa704df-kube-api-access-r7s8k\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.052809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.052697 6273 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c9168b-aa81-46fc-a51a-ec217aa704df-console-config\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:18:16.553057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.552957 6273 generic.go:356] "Generic (PLEG): container finished" podID="a7c9168b-aa81-46fc-a51a-ec217aa704df" containerID="fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b" exitCode=0 Nov 02 06:18:16.553057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.553036 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f78595c7-79l6n" Nov 02 06:18:16.553246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.553049 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f78595c7-79l6n" event={"ID":"a7c9168b-aa81-46fc-a51a-ec217aa704df","Type":"ContainerDied","Data":"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b"} Nov 02 06:18:16.553246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.553109 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f78595c7-79l6n" event={"ID":"a7c9168b-aa81-46fc-a51a-ec217aa704df","Type":"ContainerDied","Data":"d52d8b31c3970d70efef4ec4b4fa859b1fa47b54350c830c3332d289b0b52515"} Nov 02 06:18:16.553246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.553131 6273 scope.go:122] "RemoveContainer" containerID="fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b" Nov 02 06:18:16.561148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.561129 6273 scope.go:122] "RemoveContainer" containerID="fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b" Nov 02 06:18:16.561688 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:18:16.561669 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b\": container with ID starting with fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b not found: ID does not exist" containerID="fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b" Nov 02 06:18:16.561763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.561695 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b"} err="failed to get container status \"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b\": rpc error: code = NotFound desc = could not find container \"fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b\": container with ID starting with fa2a98d9d024b4410bb9ada4797a5b636c436095a7b3e1f2d9c05ee3854f036b not found: ID does not exist" Nov 02 06:18:16.706351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.706324 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:18:16.753077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:16.753031 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58f78595c7-79l6n"] Nov 02 06:18:18.176731 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:18.176696 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c9168b-aa81-46fc-a51a-ec217aa704df" path="/var/lib/kubelet/pods/a7c9168b-aa81-46fc-a51a-ec217aa704df/volumes" Nov 02 06:18:18.390586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:18.390552 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:18:18.406086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:18.406065 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:18:18.577474 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:18:18.577415 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 06:18:38.551114 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:18:38.551079 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:19:08.551255 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:19:08.551175 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:19:38.549999 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:19:38.549940 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:20:08.551739 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:20:08.551701 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:20:38.550888 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:20:38.550813 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:21:00.141415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141381 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554"] Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141709 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" containerName="console" Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141720 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" containerName="console" Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141732 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a7c9168b-aa81-46fc-a51a-ec217aa704df" containerName="console" Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141737 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c9168b-aa81-46fc-a51a-ec217aa704df" containerName="console" Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141807 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2cc1bb-2724-4b77-80a6-20be4eb4ca52" containerName="console" Nov 02 06:21:00.141833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.141815 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c9168b-aa81-46fc-a51a-ec217aa704df" containerName="console" Nov 02 06:21:00.144034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.144018 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.220819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.220793 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.220935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.220827 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.220935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.220846 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.221034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.221003 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8w5\" (UniqueName: \"kubernetes.io/projected/2ac0a724-83c4-4a37-ae43-c665b06cff08-kube-api-access-qb8w5\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.221076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.221047 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.221141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.221123 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.259214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.259195 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:00.259331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.259312 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:00.259689 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.259674 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 06:21:00.259804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.259789 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 06:21:00.260079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.260050 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 06:21:00.260079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.260068 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5hv8f\"" Nov 02 06:21:00.261544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.261521 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554"] Nov 02 06:21:00.321464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321439 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.321576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321471 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.321576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321489 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.321691 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321651 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8w5\" (UniqueName: \"kubernetes.io/projected/2ac0a724-83c4-4a37-ae43-c665b06cff08-kube-api-access-qb8w5\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.321749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321709 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.321805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.321750 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.322100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.322080 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.323885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.323860 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ac0a724-83c4-4a37-ae43-c665b06cff08-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.324069 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.324047 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.324225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.324205 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.324278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.324205 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac0a724-83c4-4a37-ae43-c665b06cff08-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.376062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.376033 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8w5\" (UniqueName: \"kubernetes.io/projected/2ac0a724-83c4-4a37-ae43-c665b06cff08-kube-api-access-qb8w5\") pod \"opendatahub-operator-controller-manager-565d9484c7-x2554\" (UID: \"2ac0a724-83c4-4a37-ae43-c665b06cff08\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.456908 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.456875 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:00.939611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.939577 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554"] Nov 02 06:21:00.947032 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:00.947002 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac0a724_83c4_4a37_ae43_c665b06cff08.slice/crio-1d11f58c8fe8ab8d7263556689d214929769dec71395960e246df73ab2ecbc29 WatchSource:0}: Error finding container 1d11f58c8fe8ab8d7263556689d214929769dec71395960e246df73ab2ecbc29: Status 404 returned error can't find the container with id 1d11f58c8fe8ab8d7263556689d214929769dec71395960e246df73ab2ecbc29 Nov 02 06:21:00.948652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:00.948635 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:21:01.152049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:01.152015 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" event={"ID":"2ac0a724-83c4-4a37-ae43-c665b06cff08","Type":"ContainerStarted","Data":"1d11f58c8fe8ab8d7263556689d214929769dec71395960e246df73ab2ecbc29"} Nov 02 06:21:07.181786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:07.181752 6273 generic.go:356] "Generic (PLEG): container finished" podID="2ac0a724-83c4-4a37-ae43-c665b06cff08" containerID="a1de8b6ae908f835f1c41ef8afc550251f3c551e4f39bb1310177b6a6633eabb" exitCode=0 Nov 02 06:21:07.182125 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:07.181831 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" event={"ID":"2ac0a724-83c4-4a37-ae43-c665b06cff08","Type":"ContainerDied","Data":"a1de8b6ae908f835f1c41ef8afc550251f3c551e4f39bb1310177b6a6633eabb"} Nov 02 06:21:08.187669 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:08.187633 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" event={"ID":"2ac0a724-83c4-4a37-ae43-c665b06cff08","Type":"ContainerStarted","Data":"7fe1b6a342a33486999c563ed804a54ebde911622ea4509884e62e975636fcc8"} Nov 02 06:21:08.188014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:08.187710 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:08.401152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:08.401110 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" podStartSLOduration=2.463041095 podStartE2EDuration="9.401098973s" totalImagesPullingTime="6.450271798s" totalInitContainerRuntime="487.78608ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:08.354395124 +0000 UTC m=+222.327729832" watchObservedRunningTime="2026-11-02 06:21:08.401098973 +0000 UTC m=+222.374433660" Nov 02 06:21:08.551280 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:21:08.551240 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:21:10.105693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.105659 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw"] Nov 02 06:21:10.108129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.108105 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.203683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.203656 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:21:10.203683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.203660 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 06:21:10.203849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.203682 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-k2fcn\"" Nov 02 06:21:10.204547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.204532 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw"] Nov 02 06:21:10.213814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.213795 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpdn\" (UniqueName: \"kubernetes.io/projected/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-kube-api-access-ttpdn\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.213865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.213849 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-tmp\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.315178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.315147 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-tmp\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.315325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.315209 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpdn\" (UniqueName: \"kubernetes.io/projected/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-kube-api-access-ttpdn\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.315535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.315517 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-tmp\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.368698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.368637 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpdn\" (UniqueName: \"kubernetes.io/projected/ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29-kube-api-access-ttpdn\") pod \"jobset-operator-674cd59dc5-2w7zw\" (UID: \"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.431122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.431097 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" Nov 02 06:21:10.799475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:10.799443 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw"] Nov 02 06:21:10.807266 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:10.807240 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef07daf0_d1d7_4abb_a97d_cbf6d3b73a29.slice/crio-800d6c626e942ded8fd9d3d1efe139d69a63df157094c2fe3640c837ea2c84c1 WatchSource:0}: Error finding container 800d6c626e942ded8fd9d3d1efe139d69a63df157094c2fe3640c837ea2c84c1: Status 404 returned error can't find the container with id 800d6c626e942ded8fd9d3d1efe139d69a63df157094c2fe3640c837ea2c84c1 Nov 02 06:21:11.200137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:11.200106 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" event={"ID":"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29","Type":"ContainerStarted","Data":"800d6c626e942ded8fd9d3d1efe139d69a63df157094c2fe3640c837ea2c84c1"} Nov 02 06:21:15.219133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:15.219059 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" event={"ID":"ef07daf0-d1d7-4abb-a97d-cbf6d3b73a29","Type":"ContainerStarted","Data":"6349f6ac266fa19a98756d0a6623e54b3ccd1c5b306243dd448daa7b6f064c97"} Nov 02 06:21:15.330984 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:15.330917 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2w7zw" podStartSLOduration=1.221382145 podStartE2EDuration="5.330899723s" totalImagesPullingTime="4.109517578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:15.330899651 +0000 UTC m=+229.304234340" watchObservedRunningTime="2026-11-02 06:21:15.330899723 +0000 UTC m=+229.304234408" Nov 02 06:21:19.194365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:19.194335 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-x2554" Nov 02 06:21:28.213908 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.213876 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577"] Nov 02 06:21:28.218298 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.218277 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.264926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.264895 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjnrd\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-kube-api-access-tjnrd\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.265060 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.264956 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.273072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.267694 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-whckf\"" Nov 02 06:21:28.313698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.313673 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 06:21:28.313698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.313697 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 06:21:28.314458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.314436 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577"] Nov 02 06:21:28.365745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.365723 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjnrd\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-kube-api-access-tjnrd\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.365873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.365776 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.420028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.420005 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.420167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.420149 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjnrd\" (UniqueName: \"kubernetes.io/projected/9c8cca55-3e95-42c6-9f83-c12c12704db1-kube-api-access-tjnrd\") pod \"cert-manager-cainjector-5c57bbd4fc-h6577\" (UID: \"9c8cca55-3e95-42c6-9f83-c12c12704db1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.529098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.529041 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" Nov 02 06:21:28.893584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:28.893515 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577"] Nov 02 06:21:28.901744 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:28.901717 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c8cca55_3e95_42c6_9f83_c12c12704db1.slice/crio-02f882c88ed35fdbfa773f1857c3123cc2cfd0e0d7efe84895f35bdfeece2ad1 WatchSource:0}: Error finding container 02f882c88ed35fdbfa773f1857c3123cc2cfd0e0d7efe84895f35bdfeece2ad1: Status 404 returned error can't find the container with id 02f882c88ed35fdbfa773f1857c3123cc2cfd0e0d7efe84895f35bdfeece2ad1 Nov 02 06:21:29.275177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.275143 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" event={"ID":"9c8cca55-3e95-42c6-9f83-c12c12704db1","Type":"ContainerStarted","Data":"02f882c88ed35fdbfa773f1857c3123cc2cfd0e0d7efe84895f35bdfeece2ad1"} Nov 02 06:21:29.315496 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.315470 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr"] Nov 02 06:21:29.318650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.318628 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.384010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.374837 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mj4ct\"" Nov 02 06:21:29.384010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.374898 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 06:21:29.384662 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.384626 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/932c90b7-2eb1-415b-b2e1-98622b054566-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.384875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.384849 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/932c90b7-2eb1-415b-b2e1-98622b054566-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.385006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.384975 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bblm\" (UniqueName: \"kubernetes.io/projected/932c90b7-2eb1-415b-b2e1-98622b054566-kube-api-access-7bblm\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.420829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.420801 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 06:21:29.420829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.420804 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:29.421008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.420831 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:29.421730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.421712 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr"] Nov 02 06:21:29.485921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.485895 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/932c90b7-2eb1-415b-b2e1-98622b054566-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.486058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.485936 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/932c90b7-2eb1-415b-b2e1-98622b054566-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.486058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.485956 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bblm\" (UniqueName: \"kubernetes.io/projected/932c90b7-2eb1-415b-b2e1-98622b054566-kube-api-access-7bblm\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.486646 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.486627 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/932c90b7-2eb1-415b-b2e1-98622b054566-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.488360 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.488338 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/932c90b7-2eb1-415b-b2e1-98622b054566-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.539789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.539736 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bblm\" (UniqueName: \"kubernetes.io/projected/932c90b7-2eb1-415b-b2e1-98622b054566-kube-api-access-7bblm\") pod \"kuadrant-console-plugin-66c4fb575d-6ngfr\" (UID: \"932c90b7-2eb1-415b-b2e1-98622b054566\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:29.629036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:29.629015 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" Nov 02 06:21:30.000508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:30.000473 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr"] Nov 02 06:21:30.008500 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:30.008466 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod932c90b7_2eb1_415b_b2e1_98622b054566.slice/crio-fe4cb3e732302e9ee76b24c08ce4ba4737016d437147e0206f8424c7add657be WatchSource:0}: Error finding container fe4cb3e732302e9ee76b24c08ce4ba4737016d437147e0206f8424c7add657be: Status 404 returned error can't find the container with id fe4cb3e732302e9ee76b24c08ce4ba4737016d437147e0206f8424c7add657be Nov 02 06:21:30.280296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:30.280224 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" event={"ID":"932c90b7-2eb1-415b-b2e1-98622b054566","Type":"ContainerStarted","Data":"fe4cb3e732302e9ee76b24c08ce4ba4737016d437147e0206f8424c7add657be"} Nov 02 06:21:32.022430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.022394 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx"] Nov 02 06:21:32.025348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.025321 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.074939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.074910 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qshtf\"" Nov 02 06:21:32.078664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.078639 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx"] Nov 02 06:21:32.110363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.110332 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446lf\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-kube-api-access-446lf\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.110514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.110445 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.211183 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.211154 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446lf\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-kube-api-access-446lf\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.211340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.211207 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.266476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.266447 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446lf\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-kube-api-access-446lf\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.267452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.267421 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b465618f-6a18-431a-956a-61f3fb5e34bd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwhx\" (UID: \"b465618f-6a18-431a-956a-61f3fb5e34bd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.339048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.338941 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:32.719078 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:32.719043 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx"] Nov 02 06:21:32.728512 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:32.728482 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb465618f_6a18_431a_956a_61f3fb5e34bd.slice/crio-e9104aeaab2ca7d60dccd6e3eb367cdf693ffc926c0b94c88d6740ca858c30c2 WatchSource:0}: Error finding container e9104aeaab2ca7d60dccd6e3eb367cdf693ffc926c0b94c88d6740ca858c30c2: Status 404 returned error can't find the container with id e9104aeaab2ca7d60dccd6e3eb367cdf693ffc926c0b94c88d6740ca858c30c2 Nov 02 06:21:33.294487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:33.294454 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" event={"ID":"b465618f-6a18-431a-956a-61f3fb5e34bd","Type":"ContainerStarted","Data":"e9104aeaab2ca7d60dccd6e3eb367cdf693ffc926c0b94c88d6740ca858c30c2"} Nov 02 06:21:35.553160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.551364 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-7wznk"] Nov 02 06:21:35.561712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.560313 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.609084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.608424 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-7wznk"] Nov 02 06:21:35.644539 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.644511 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tmp\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.644702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.644545 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pfx\" (UniqueName: \"kubernetes.io/projected/af32b8a2-f1c4-4adc-9883-de2d21a64c43-kube-api-access-s2pfx\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.644702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.644634 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tls-certs\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.654433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.654368 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 06:21:35.654433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.654376 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 06:21:35.654656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.654474 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 06:21:35.663150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.663026 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g6qjm\"" Nov 02 06:21:35.663672 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.663555 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 06:21:35.745257 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.745222 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tmp\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.745442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.745270 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pfx\" (UniqueName: \"kubernetes.io/projected/af32b8a2-f1c4-4adc-9883-de2d21a64c43-kube-api-access-s2pfx\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.745442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.745322 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tls-certs\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.748553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.748526 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tls-certs\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.750266 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.750239 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af32b8a2-f1c4-4adc-9883-de2d21a64c43-tmp\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.811458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.811392 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pfx\" (UniqueName: \"kubernetes.io/projected/af32b8a2-f1c4-4adc-9883-de2d21a64c43-kube-api-access-s2pfx\") pod \"kube-auth-proxy-784594468b-7wznk\" (UID: \"af32b8a2-f1c4-4adc-9883-de2d21a64c43\") " pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:35.881859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:35.881645 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" Nov 02 06:21:36.280182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:36.280136 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-7wznk"] Nov 02 06:21:36.290803 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:36.290769 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf32b8a2_f1c4_4adc_9883_de2d21a64c43.slice/crio-086ec9cdec8dd0e2ea74aab6ea49f9c82de69ecd767300cb7fad04a5f5faf1ca WatchSource:0}: Error finding container 086ec9cdec8dd0e2ea74aab6ea49f9c82de69ecd767300cb7fad04a5f5faf1ca: Status 404 returned error can't find the container with id 086ec9cdec8dd0e2ea74aab6ea49f9c82de69ecd767300cb7fad04a5f5faf1ca Nov 02 06:21:36.315584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:36.315448 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" event={"ID":"af32b8a2-f1c4-4adc-9883-de2d21a64c43","Type":"ContainerStarted","Data":"086ec9cdec8dd0e2ea74aab6ea49f9c82de69ecd767300cb7fad04a5f5faf1ca"} Nov 02 06:21:37.321445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:37.321401 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" event={"ID":"b465618f-6a18-431a-956a-61f3fb5e34bd","Type":"ContainerStarted","Data":"406b44cf9699e1a1ecddd9771d8838989dddbb0d84a3446b578682a62dbb56ea"} Nov 02 06:21:37.321867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:37.321581 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:37.323061 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:37.323029 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" event={"ID":"9c8cca55-3e95-42c6-9f83-c12c12704db1","Type":"ContainerStarted","Data":"0bf2830ba14ac2401e6967343a4c5a7113722dd0beb3540db4d3cdf74ec6ba06"} Nov 02 06:21:37.429032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:37.428961 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" podStartSLOduration=2.263905894 podStartE2EDuration="6.428947229s" totalImagesPullingTime="4.165041335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:37.427484754 +0000 UTC m=+251.400819475" watchObservedRunningTime="2026-11-02 06:21:37.428947229 +0000 UTC m=+251.402281906" Nov 02 06:21:37.543915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:37.543847 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h6577" podStartSLOduration=2.139848965 podStartE2EDuration="9.543828331s" totalImagesPullingTime="7.403979366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:37.543216302 +0000 UTC m=+251.516550988" watchObservedRunningTime="2026-11-02 06:21:37.543828331 +0000 UTC m=+251.517163016" Nov 02 06:21:38.328696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:38.328603 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" event={"ID":"932c90b7-2eb1-415b-b2e1-98622b054566","Type":"ContainerStarted","Data":"1baf12c91d2446015ad937b28e10beeb10bcdb642bf187eaf8ab2a2d832ee74b"} Nov 02 06:21:38.436437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:38.436387 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6ngfr" podStartSLOduration=1.519556742 podStartE2EDuration="9.436374662s" totalImagesPullingTime="7.91681792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:38.436162668 +0000 UTC m=+252.409497349" watchObservedRunningTime="2026-11-02 06:21:38.436374662 +0000 UTC m=+252.409709373" Nov 02 06:21:38.555528 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:21:38.555491 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:21:39.789616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.789584 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:21:39.794212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.794190 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:39.843269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.843241 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 06:21:39.887106 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.887084 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdfl\" (UniqueName: \"kubernetes.io/projected/af82ea08-8e55-418c-a817-426f7e8721f1-kube-api-access-xxdfl\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:39.887199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.887136 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af82ea08-8e55-418c-a817-426f7e8721f1-config-file\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:39.889953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.889928 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:21:39.988016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.987955 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdfl\" (UniqueName: \"kubernetes.io/projected/af82ea08-8e55-418c-a817-426f7e8721f1-kube-api-access-xxdfl\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:39.988189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.988100 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af82ea08-8e55-418c-a817-426f7e8721f1-config-file\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:39.988816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:39.988794 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af82ea08-8e55-418c-a817-426f7e8721f1-config-file\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:40.041706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.041648 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdfl\" (UniqueName: \"kubernetes.io/projected/af82ea08-8e55-418c-a817-426f7e8721f1-kube-api-access-xxdfl\") pod \"limitador-limitador-7c9b87d574-n2zvg\" (UID: \"af82ea08-8e55-418c-a817-426f7e8721f1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:40.108454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.108429 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:40.123613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.123590 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:21:40.339404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.339321 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" event={"ID":"af32b8a2-f1c4-4adc-9883-de2d21a64c43","Type":"ContainerStarted","Data":"7bba303c248fbc9a4469fd56a24b07afa0c54a727213b72d55dc035f3cb48dfd"} Nov 02 06:21:40.449080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.449028 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-784594468b-7wznk" podStartSLOduration=1.877969249 podStartE2EDuration="5.449015737s" totalImagesPullingTime="3.571046488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:40.448628503 +0000 UTC m=+254.421963185" watchObservedRunningTime="2026-11-02 06:21:40.449015737 +0000 UTC m=+254.422350417" Nov 02 06:21:40.483896 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:40.483867 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82ea08_8e55_418c_a817_426f7e8721f1.slice/crio-e050f9ad029eeed0ad3b764f1fda912055e5be3afab2fcf686c1d4e4d0443076 WatchSource:0}: Error finding container e050f9ad029eeed0ad3b764f1fda912055e5be3afab2fcf686c1d4e4d0443076: Status 404 returned error can't find the container with id e050f9ad029eeed0ad3b764f1fda912055e5be3afab2fcf686c1d4e4d0443076 Nov 02 06:21:40.495870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:40.495849 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:21:41.344389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:41.344351 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" event={"ID":"af82ea08-8e55-418c-a817-426f7e8721f1","Type":"ContainerStarted","Data":"e050f9ad029eeed0ad3b764f1fda912055e5be3afab2fcf686c1d4e4d0443076"} Nov 02 06:21:43.331597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:43.331567 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" Nov 02 06:21:46.368244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:46.368209 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" event={"ID":"af82ea08-8e55-418c-a817-426f7e8721f1","Type":"ContainerStarted","Data":"b68cc92aaf61e6a45c8dbaac900e94ddfe256a1fab5fa40a690d256bf3d15658"} Nov 02 06:21:46.368832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:46.368464 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:21:46.476361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:46.476304 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" podStartSLOduration=2.1701026150000002 podStartE2EDuration="7.476287671s" totalImagesPullingTime="5.306185056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:46.47533302 +0000 UTC m=+260.448667704" watchObservedRunningTime="2026-11-02 06:21:46.476287671 +0000 UTC m=+260.449622357" Nov 02 06:21:47.262745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.262708 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv"] Nov 02 06:21:47.266291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.266266 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.315696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.315675 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:21:47.350032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.350005 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh59b\" (UniqueName: \"kubernetes.io/projected/97c0055c-9928-4129-a6f2-34ce3cf361f6-kube-api-access-xh59b\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.350141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.350057 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.350190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.350146 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97c0055c-9928-4129-a6f2-34ce3cf361f6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.350234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.350207 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.361740 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.361718 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:21:47.361831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.361769 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pxm9k\"" Nov 02 06:21:47.451077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451044 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.451529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451095 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97c0055c-9928-4129-a6f2-34ce3cf361f6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.451529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451136 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.451529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451187 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh59b\" (UniqueName: \"kubernetes.io/projected/97c0055c-9928-4129-a6f2-34ce3cf361f6-kube-api-access-xh59b\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.451529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451451 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.451683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.451532 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97c0055c-9928-4129-a6f2-34ce3cf361f6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.455512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.455483 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 06:21:47.456803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.456786 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv"] Nov 02 06:21:47.462297 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.462278 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97c0055c-9928-4129-a6f2-34ce3cf361f6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.505337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.505310 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh59b\" (UniqueName: \"kubernetes.io/projected/97c0055c-9928-4129-a6f2-34ce3cf361f6-kube-api-access-xh59b\") pod \"kserve-module-controller-manager-7566cf5fc8-8fqnv\" (UID: \"97c0055c-9928-4129-a6f2-34ce3cf361f6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:47.578346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:47.578270 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:48.011463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:48.008937 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv"] Nov 02 06:21:48.022093 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:21:48.022061 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c0055c_9928_4129_a6f2_34ce3cf361f6.slice/crio-9aa9305f7c5ff96b1d2171bf7ccdc2a2e3af4ed0d908afb6042cfb4cab23b768 WatchSource:0}: Error finding container 9aa9305f7c5ff96b1d2171bf7ccdc2a2e3af4ed0d908afb6042cfb4cab23b768: Status 404 returned error can't find the container with id 9aa9305f7c5ff96b1d2171bf7ccdc2a2e3af4ed0d908afb6042cfb4cab23b768 Nov 02 06:21:48.377613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:48.377537 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" event={"ID":"97c0055c-9928-4129-a6f2-34ce3cf361f6","Type":"ContainerStarted","Data":"9aa9305f7c5ff96b1d2171bf7ccdc2a2e3af4ed0d908afb6042cfb4cab23b768"} Nov 02 06:21:51.393133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:51.393097 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" event={"ID":"97c0055c-9928-4129-a6f2-34ce3cf361f6","Type":"ContainerStarted","Data":"b78c5dfe1e8429c2b4f6e960240f28b6748c1ca8279ad7dcd02b3df387e02065"} Nov 02 06:21:51.393563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:51.393216 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:21:51.548327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:51.548274 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" podStartSLOduration=1.9716524359999998 podStartE2EDuration="4.548260314s" totalImagesPullingTime="2.576607878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:51.547442765 +0000 UTC m=+265.520777446" watchObservedRunningTime="2026-11-02 06:21:51.548260314 +0000 UTC m=+265.521594998" Nov 02 06:21:57.373344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:21:57.373313 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n2zvg" Nov 02 06:22:08.554378 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:22:08.554297 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:22:12.398500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:12.398469 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8fqnv" Nov 02 06:22:38.551864 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:22:38.551823 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:22:51.281590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.281554 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-67qww"] Nov 02 06:22:51.288509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.288487 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.317512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.317487 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86ln\" (UniqueName: \"kubernetes.io/projected/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-kube-api-access-r86ln\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.317626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.317575 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-tls-certs\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.339778 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.339755 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-67qww"] Nov 02 06:22:51.384382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.384354 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9k4hz\"" Nov 02 06:22:51.384382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.384374 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 06:22:51.385459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.385435 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2b59c"] Nov 02 06:22:51.389006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.388967 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.418312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.418292 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-cert\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.418415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.418340 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-tls-certs\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.418415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.418370 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjlz\" (UniqueName: \"kubernetes.io/projected/53fcf074-8bd9-42be-8c55-c41317dfa9fa-kube-api-access-6hjlz\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.418520 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.418449 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-metrics-certs\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.418520 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.418495 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86ln\" (UniqueName: \"kubernetes.io/projected/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-kube-api-access-r86ln\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.420724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.420706 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-tls-certs\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.437120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.437099 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2b59c"] Nov 02 06:22:51.438366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.438345 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-krgkm\"" Nov 02 06:22:51.438366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.438349 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 06:22:51.438503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.438417 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 06:22:51.475200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.475180 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86ln\" (UniqueName: \"kubernetes.io/projected/5b5a208a-14d3-4fd1-adac-c6e4be0a27e9-kube-api-access-r86ln\") pod \"model-serving-api-75bd89fdfc-67qww\" (UID: \"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9\") " pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.519255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.519227 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-cert\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.519404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.519276 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjlz\" (UniqueName: \"kubernetes.io/projected/53fcf074-8bd9-42be-8c55-c41317dfa9fa-kube-api-access-6hjlz\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.519404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.519297 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-metrics-certs\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.521545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.521523 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-metrics-certs\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.521735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.521712 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53fcf074-8bd9-42be-8c55-c41317dfa9fa-cert\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.572432 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.572374 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjlz\" (UniqueName: \"kubernetes.io/projected/53fcf074-8bd9-42be-8c55-c41317dfa9fa-kube-api-access-6hjlz\") pod \"odh-model-controller-769875db56-2b59c\" (UID: \"53fcf074-8bd9-42be-8c55-c41317dfa9fa\") " pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.601320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.601301 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:51.701146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.701117 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:51.966756 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:51.966725 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-67qww"] Nov 02 06:22:51.974286 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:22:51.974260 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5a208a_14d3_4fd1_adac_c6e4be0a27e9.slice/crio-311cc5e01103a3279a30035ab5f192eb52ce075941733078c44af575cb8bb3b9 WatchSource:0}: Error finding container 311cc5e01103a3279a30035ab5f192eb52ce075941733078c44af575cb8bb3b9: Status 404 returned error can't find the container with id 311cc5e01103a3279a30035ab5f192eb52ce075941733078c44af575cb8bb3b9 Nov 02 06:22:52.082161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:52.082130 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2b59c"] Nov 02 06:22:52.089745 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:22:52.089710 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fcf074_8bd9_42be_8c55_c41317dfa9fa.slice/crio-1ce7d600f960cb926c135baaaad4533c6edd3a81e979b0686bdcfba03a0e6c82 WatchSource:0}: Error finding container 1ce7d600f960cb926c135baaaad4533c6edd3a81e979b0686bdcfba03a0e6c82: Status 404 returned error can't find the container with id 1ce7d600f960cb926c135baaaad4533c6edd3a81e979b0686bdcfba03a0e6c82 Nov 02 06:22:52.652952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:52.652909 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2b59c" event={"ID":"53fcf074-8bd9-42be-8c55-c41317dfa9fa","Type":"ContainerStarted","Data":"1ce7d600f960cb926c135baaaad4533c6edd3a81e979b0686bdcfba03a0e6c82"} Nov 02 06:22:52.654090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:52.654064 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" event={"ID":"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9","Type":"ContainerStarted","Data":"311cc5e01103a3279a30035ab5f192eb52ce075941733078c44af575cb8bb3b9"} Nov 02 06:22:55.670222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:55.670186 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" event={"ID":"5b5a208a-14d3-4fd1-adac-c6e4be0a27e9","Type":"ContainerStarted","Data":"5e683d01fae4d22e7ab793e450e73befef4ae977c74d9b8acf6528b136d72394"} Nov 02 06:22:55.670627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:55.670436 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:22:55.779943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:55.779894 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" podStartSLOduration=1.706317187 podStartE2EDuration="4.77988021s" totalImagesPullingTime="3.073563023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:55.778902045 +0000 UTC m=+329.752236730" watchObservedRunningTime="2026-11-02 06:22:55.77988021 +0000 UTC m=+329.753214893" Nov 02 06:22:57.679961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:57.679919 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2b59c" event={"ID":"53fcf074-8bd9-42be-8c55-c41317dfa9fa","Type":"ContainerStarted","Data":"ea5163b5654fcecfb2dabecb1db9bbc79f00b6b8b4619f34b595562ddbe18f59"} Nov 02 06:22:57.680511 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:57.680064 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:22:57.789942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:22:57.789878 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2b59c" podStartSLOduration=1.5308058359999999 podStartE2EDuration="6.789862348s" totalImagesPullingTime="5.259056512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:57.789182412 +0000 UTC m=+331.762517095" watchObservedRunningTime="2026-11-02 06:22:57.789862348 +0000 UTC m=+331.763197032" Nov 02 06:23:06.678196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:06.678168 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-67qww" Nov 02 06:23:08.552746 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:23:08.552703 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:23:08.687065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:08.687036 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2b59c" Nov 02 06:23:18.671880 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.671846 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c"] Nov 02 06:23:18.675277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.675253 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.724519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.724494 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 06:23:18.770663 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.770635 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 06:23:18.770856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.770833 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 06:23:18.770921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.770864 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-q8pl8\"" Nov 02 06:23:18.772536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.772517 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c"] Nov 02 06:23:18.776718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.776699 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-cert\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.776816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.776752 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsgx\" (UniqueName: \"kubernetes.io/projected/699cd8ab-2763-4009-9b89-168f4be4ad68-kube-api-access-dhsgx\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.776861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.776842 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/699cd8ab-2763-4009-9b89-168f4be4ad68-manager-config\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.776903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.776864 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-metrics-certs\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.878362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.878329 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/699cd8ab-2763-4009-9b89-168f4be4ad68-manager-config\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.878362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.878364 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-metrics-certs\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.878557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.878398 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-cert\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.878557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.878447 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsgx\" (UniqueName: \"kubernetes.io/projected/699cd8ab-2763-4009-9b89-168f4be4ad68-kube-api-access-dhsgx\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.879090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.879068 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/699cd8ab-2763-4009-9b89-168f4be4ad68-manager-config\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.880769 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.880743 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-metrics-certs\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.880922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.880901 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/699cd8ab-2763-4009-9b89-168f4be4ad68-cert\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.931587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.931534 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsgx\" (UniqueName: \"kubernetes.io/projected/699cd8ab-2763-4009-9b89-168f4be4ad68-kube-api-access-dhsgx\") pod \"jobset-controller-manager-fd9787bf4-26l7c\" (UID: \"699cd8ab-2763-4009-9b89-168f4be4ad68\") " pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:18.987227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:18.987203 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:19.354497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:19.354422 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c"] Nov 02 06:23:19.361759 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:23:19.361726 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699cd8ab_2763_4009_9b89_168f4be4ad68.slice/crio-5c718ff0327bac278338e13659a10f98cf5030a84c69ec225bb6b78e8dc7d8ba WatchSource:0}: Error finding container 5c718ff0327bac278338e13659a10f98cf5030a84c69ec225bb6b78e8dc7d8ba: Status 404 returned error can't find the container with id 5c718ff0327bac278338e13659a10f98cf5030a84c69ec225bb6b78e8dc7d8ba Nov 02 06:23:19.780623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:19.780584 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" event={"ID":"699cd8ab-2763-4009-9b89-168f4be4ad68","Type":"ContainerStarted","Data":"5c718ff0327bac278338e13659a10f98cf5030a84c69ec225bb6b78e8dc7d8ba"} Nov 02 06:23:19.854973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:19.854941 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:23:24.804070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:24.804034 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" event={"ID":"699cd8ab-2763-4009-9b89-168f4be4ad68","Type":"ContainerStarted","Data":"d307ecd07d78d4dbfa74eca6482d1c8e03dc617f3b5606cb2b24c9b60a0d713c"} Nov 02 06:23:24.804464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:24.804181 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:24.911682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:24.911619 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" podStartSLOduration=1.919631581 podStartE2EDuration="6.911605151s" totalImagesPullingTime="4.99197357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:24.910702309 +0000 UTC m=+358.884036993" watchObservedRunningTime="2026-11-02 06:23:24.911605151 +0000 UTC m=+358.884939835" Nov 02 06:23:35.813350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:23:35.813276 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-fd9787bf4-26l7c" Nov 02 06:23:38.552551 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:23:38.552514 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:24:08.552401 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:24:08.552359 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:24:38.552604 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:24:38.552568 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:25:08.552301 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:25:08.552205 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:25:11.266530 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:11.266498 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:25:20.327967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:20.327932 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:25:30.215977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:30.215946 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:25:33.570537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.570507 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wfdqg"] Nov 02 06:25:33.574237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.574215 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.669642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.669602 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 06:25:33.669809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.669692 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 06:25:33.669809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.669700 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lxzd7\"" Nov 02 06:25:33.669809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.669754 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 06:25:33.670998 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.670968 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wfdqg"] Nov 02 06:25:33.726493 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726466 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnt6\" (UniqueName: \"kubernetes.io/projected/ffe42dde-6598-48cf-9eb6-351ed0edb27b-kube-api-access-jlnt6\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.726622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726497 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.726622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726521 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.726709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726652 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.726709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726695 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.726807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.726721 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.827978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.827904 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.827978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.827965 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828011 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828032 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnt6\" (UniqueName: \"kubernetes.io/projected/ffe42dde-6598-48cf-9eb6-351ed0edb27b-kube-api-access-jlnt6\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828057 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828095 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828381 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828486 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828465 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-home\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.828550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.828535 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.830284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.830256 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe42dde-6598-48cf-9eb6-351ed0edb27b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.830565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.830545 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe42dde-6598-48cf-9eb6-351ed0edb27b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.880875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.880852 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnt6\" (UniqueName: \"kubernetes.io/projected/ffe42dde-6598-48cf-9eb6-351ed0edb27b-kube-api-access-jlnt6\") pod \"echo-simulator-kserve-64b6f8f96c-wfdqg\" (UID: \"ffe42dde-6598-48cf-9eb6-351ed0edb27b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:33.890715 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:33.890695 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:34.253661 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:34.253629 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-wfdqg"] Nov 02 06:25:34.257371 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:25:34.257342 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe42dde_6598_48cf_9eb6_351ed0edb27b.slice/crio-654c0936d7969556d978d50ae35f020f14956d2ac0d1035034fe7e0ecc5cadc4 WatchSource:0}: Error finding container 654c0936d7969556d978d50ae35f020f14956d2ac0d1035034fe7e0ecc5cadc4: Status 404 returned error can't find the container with id 654c0936d7969556d978d50ae35f020f14956d2ac0d1035034fe7e0ecc5cadc4 Nov 02 06:25:34.403120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:34.403089 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" event={"ID":"ffe42dde-6598-48cf-9eb6-351ed0edb27b","Type":"ContainerStarted","Data":"654c0936d7969556d978d50ae35f020f14956d2ac0d1035034fe7e0ecc5cadc4"} Nov 02 06:25:37.417760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:37.417729 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" event={"ID":"ffe42dde-6598-48cf-9eb6-351ed0edb27b","Type":"ContainerStarted","Data":"b64471811ca25eab3a439ab826d9402bf3686eb5443c971a14a363afa8562f05"} Nov 02 06:25:37.525439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:37.525390 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" podStartSLOduration=1.950033351 podStartE2EDuration="4.525374978s" totalImagesPullingTime="2.575341627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:37.524244694 +0000 UTC m=+491.497579375" watchObservedRunningTime="2026-11-02 06:25:37.525374978 +0000 UTC m=+491.498709662" Nov 02 06:25:37.891848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:37.891765 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:38.551384 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:25:38.551350 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:25:43.891624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:43.891588 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:43.904188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:43.904164 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:44.460029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:44.459983 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-wfdqg" Nov 02 06:25:45.668806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.668766 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:25:45.672715 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.672689 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.715262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.715233 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:25:45.718579 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.718560 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.730007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.729972 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 06:25:45.730238 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.730220 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:25:45.769447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.769424 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:25:45.815316 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.815285 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:25:45.838921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.838895 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.838941 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839063 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839138 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839173 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839209 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839236 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839302 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839402 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839432 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839450 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839489 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839605 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zt2n\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839670 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839711 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.839796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839776 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839815 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.839883 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.839840 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940562 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940592 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940611 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940626 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940652 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zt2n\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940673 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940695 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940711 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940740 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940763 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940817 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.940884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940867 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940914 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940945 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.940979 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941043 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941071 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941076 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941096 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941205 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941260 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941389 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941466 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941655 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.941827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941813 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.941827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941825 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.942053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.941885 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.942159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.942140 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.943796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.943772 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.943912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.943890 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.944036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.944019 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.944101 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.944078 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.996721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.996697 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.996843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.996717 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:45.996843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.996760 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zt2n\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fnwg8\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:45.996973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:45.996874 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-htmtx\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:46.030656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:46.030637 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:46.285597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:46.285532 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:46.442650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:46.442619 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:25:46.446174 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:25:46.446143 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855763ec_df91_40f7_a0c2_3059e2ea2c24.slice/crio-a5c6d8eb116ab17131b6c5c547c0557cd724acf760b3d4d4d8e3166f3f661a19 WatchSource:0}: Error finding container a5c6d8eb116ab17131b6c5c547c0557cd724acf760b3d4d4d8e3166f3f661a19: Status 404 returned error can't find the container with id a5c6d8eb116ab17131b6c5c547c0557cd724acf760b3d4d4d8e3166f3f661a19 Nov 02 06:25:46.460574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:46.460548 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" event={"ID":"855763ec-df91-40f7-a0c2-3059e2ea2c24","Type":"ContainerStarted","Data":"a5c6d8eb116ab17131b6c5c547c0557cd724acf760b3d4d4d8e3166f3f661a19"} Nov 02 06:25:46.656675 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:46.656535 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:25:46.659297 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:25:46.659270 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96141757_259e_4bbd_8615_884a8edd04af.slice/crio-50f03679ad098825fedcc57283b98e985cce85dc8c4726bc92f3be564899aa01 WatchSource:0}: Error finding container 50f03679ad098825fedcc57283b98e985cce85dc8c4726bc92f3be564899aa01: Status 404 returned error can't find the container with id 50f03679ad098825fedcc57283b98e985cce85dc8c4726bc92f3be564899aa01 Nov 02 06:25:47.466114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:47.466081 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" event={"ID":"96141757-259e-4bbd-8615-884a8edd04af","Type":"ContainerStarted","Data":"50f03679ad098825fedcc57283b98e985cce85dc8c4726bc92f3be564899aa01"} Nov 02 06:25:49.631791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:49.631753 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:25:55.473581 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.473540 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:55.474037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.473630 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:55.474037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.473679 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:55.567874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.567836 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:55.568029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.567907 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:55.568029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:55.567938 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:56.511223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:56.511166 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" event={"ID":"96141757-259e-4bbd-8615-884a8edd04af","Type":"ContainerStarted","Data":"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454"} Nov 02 06:25:56.512642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:56.512616 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" event={"ID":"855763ec-df91-40f7-a0c2-3059e2ea2c24","Type":"ContainerStarted","Data":"02c834c6b4e2b69e9562cc156c78d2ecaaca0bb8bc162ad585444abf0343c7cf"} Nov 02 06:25:56.724195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:56.724138 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podStartSLOduration=2.8183303029999998 podStartE2EDuration="11.724121748s" totalImagesPullingTime="8.905791445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:56.72284589 +0000 UTC m=+510.696180585" watchObservedRunningTime="2026-11-02 06:25:56.724121748 +0000 UTC m=+510.697456430" Nov 02 06:25:57.031041 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.030980 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:25:57.032271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.032241 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:25:57.032411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.032292 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:25:57.292132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.292048 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:25:57.298160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.298121 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:25:57.298286 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:57.298196 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:25:58.031483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:58.031451 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:25:58.031885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:58.031503 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:25:58.298178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:58.298078 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:25:58.298178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:58.298142 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:25:59.031231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:59.031178 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:25:59.031410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:59.031244 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:25:59.288066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:59.286402 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:25:59.288066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:25:59.286485 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:26:00.031421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:00.031389 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:26:00.031652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:00.031629 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:26:00.290066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:00.289446 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:00.290066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:00.289524 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:26:01.030900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.030794 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:26:01.031505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.031484 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:26:01.031564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.031525 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:26:01.289068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.288073 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:26:01.289249 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.289150 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:01.289249 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:01.289197 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:26:02.030957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:02.030921 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:26:02.031372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:02.031003 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:26:02.286564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:02.286471 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:02.286564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:02.286550 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:26:03.031600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:03.031558 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:26:03.032140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:03.031637 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 06:26:03.286421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:03.286331 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:03.286421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:03.286397 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 06:26:04.036719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:04.036689 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:26:04.038011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:04.037970 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:26:04.204777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:04.204717 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podStartSLOduration=10.180432755 podStartE2EDuration="19.204700616s" totalImagesPullingTime="9.024267861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:56.842264178 +0000 UTC m=+510.815598893" watchObservedRunningTime="2026-11-02 06:26:04.204700616 +0000 UTC m=+518.178035301" Nov 02 06:26:04.289594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:04.289522 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:26:04.290515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:04.290498 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:26:08.551723 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:26:08.551686 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:26:21.059739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.059705 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-85b9b944cd-5tx47"] Nov 02 06:26:21.065696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.065679 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.121624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.121597 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhkn\" (UniqueName: \"kubernetes.io/projected/0bc2fcaa-2adb-4c3b-8300-46456515a594-kube-api-access-9dhkn\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.121794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.121647 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bc2fcaa-2adb-4c3b-8300-46456515a594-metrics-certs\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.163603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.163576 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 06:26:21.163723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.163677 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-d9228\"" Nov 02 06:26:21.163766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.163750 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 06:26:21.164950 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.164931 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85b9b944cd-5tx47"] Nov 02 06:26:21.222531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.222507 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhkn\" (UniqueName: \"kubernetes.io/projected/0bc2fcaa-2adb-4c3b-8300-46456515a594-kube-api-access-9dhkn\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.222627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.222550 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bc2fcaa-2adb-4c3b-8300-46456515a594-metrics-certs\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.224981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.224955 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bc2fcaa-2adb-4c3b-8300-46456515a594-metrics-certs\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.276726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.276696 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhkn\" (UniqueName: \"kubernetes.io/projected/0bc2fcaa-2adb-4c3b-8300-46456515a594-kube-api-access-9dhkn\") pod \"ai-gateway-controller-85b9b944cd-5tx47\" (UID: \"0bc2fcaa-2adb-4c3b-8300-46456515a594\") " pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.378552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.378499 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:21.746523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.746490 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85b9b944cd-5tx47"] Nov 02 06:26:21.750040 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:26:21.750006 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc2fcaa_2adb_4c3b_8300_46456515a594.slice/crio-26eca1fa8eddef8f51414e61b3bcc3775e2c64bebca29b04c4608cb82629cb81 WatchSource:0}: Error finding container 26eca1fa8eddef8f51414e61b3bcc3775e2c64bebca29b04c4608cb82629cb81: Status 404 returned error can't find the container with id 26eca1fa8eddef8f51414e61b3bcc3775e2c64bebca29b04c4608cb82629cb81 Nov 02 06:26:21.751385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:21.751369 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:26:22.637775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:22.637741 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" event={"ID":"0bc2fcaa-2adb-4c3b-8300-46456515a594","Type":"ContainerStarted","Data":"26eca1fa8eddef8f51414e61b3bcc3775e2c64bebca29b04c4608cb82629cb81"} Nov 02 06:26:26.187162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:26.187139 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 06:26:26.660356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:26.660314 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" event={"ID":"0bc2fcaa-2adb-4c3b-8300-46456515a594","Type":"ContainerStarted","Data":"47d03cb1f48d43989e87022cad4efa799b2bc1f335791c81dfba738da58b098d"} Nov 02 06:26:26.660532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:26.660447 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:26.768596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:26.768543 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" podStartSLOduration=1.381246666 podStartE2EDuration="5.768527665s" totalImagesPullingTime="4.387280999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:26.767662438 +0000 UTC m=+540.740997119" watchObservedRunningTime="2026-11-02 06:26:26.768527665 +0000 UTC m=+540.741862350" Nov 02 06:26:37.667614 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:26:37.667538 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-85b9b944cd-5tx47" Nov 02 06:26:38.553168 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:26:38.553134 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:27:08.551859 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:27:08.551815 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:27:11.519943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.519909 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:27:11.523934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.523913 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574480 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574569 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574679 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574751 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ggn\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574771 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574828 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574866 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574895 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.574983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.574958 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.619302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.619272 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:27:11.664231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.664210 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cdfce9-openshift-default-dockercfg-j4bz4\"" Nov 02 06:27:11.676232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676195 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ggn\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676240 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676351 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676390 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676412 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676505 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676585 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676552 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676583 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676605 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676690 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676712 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676759 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.676922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.676872 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.677191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.677169 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.678817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.678794 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.678941 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.678922 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.729584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.729562 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ggn\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.729841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.729823 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:11.836411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:11.836335 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:12.116934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.116853 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:27:12.122590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.122566 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.172095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.172071 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bdcd80-openshift-default-dockercfg-h4jqn\"" Nov 02 06:27:12.179553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.179521 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:27:12.181844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.181817 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7km\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.181981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.181865 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182042 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182075 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182101 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182228 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182277 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182360 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.182417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.182387 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.271557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.271427 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:27:12.274524 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:12.274497 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3286ad65_78af_4cda_b1fa_aae9ee36ed56.slice/crio-a08fd5491a1a2214f8c8ec599c711514d58b984f15a6f32b493de3679ca26e7a WatchSource:0}: Error finding container a08fd5491a1a2214f8c8ec599c711514d58b984f15a6f32b493de3679ca26e7a: Status 404 returned error can't find the container with id a08fd5491a1a2214f8c8ec599c711514d58b984f15a6f32b493de3679ca26e7a Nov 02 06:27:12.276768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.276737 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.276834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.276798 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.276886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.276842 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.282879 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.282855 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.282951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.282893 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.282951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.282934 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.282957 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283034 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7km\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283076 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283162 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283199 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283232 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283440 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283732 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.283847 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283763 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.284026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.283976 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.284129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.284108 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.285318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.285286 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.286331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.286308 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.337407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.337381 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7km\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.337538 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.337516 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.405703 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.405670 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:27:12.409340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.409314 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.437533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.437505 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:12.458718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.458696 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fb1398-openshift-default-dockercfg-rn84l\"" Nov 02 06:27:12.467117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.467083 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485137 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485214 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485255 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485285 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485335 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtpl\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485417 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485451 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485474 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.486599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.485512 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586405 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586453 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586581 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586643 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586718 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586780 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586798 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586821 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586829 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.586878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586849 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.587391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.586918 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtpl\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.587391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.587173 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.587391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.587341 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.587536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.587460 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.589313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.589286 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.589477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.589460 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.640569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.640532 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.641257 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.641237 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtpl\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.723168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.723106 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:12.822261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.822223 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:27:12.824430 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:12.824397 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23204de3_4d1e_41d6_b50e_cce9218a90af.slice/crio-6e807a11475874d2bb97163c52f9e5c6c829381b0bf89e63bdde5bac855f5c17 WatchSource:0}: Error finding container 6e807a11475874d2bb97163c52f9e5c6c829381b0bf89e63bdde5bac855f5c17: Status 404 returned error can't find the container with id 6e807a11475874d2bb97163c52f9e5c6c829381b0bf89e63bdde5bac855f5c17 Nov 02 06:27:12.826728 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.826694 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.826829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.826771 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.826829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.826816 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.871036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.871002 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerStarted","Data":"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6"} Nov 02 06:27:12.871159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.871073 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerStarted","Data":"a08fd5491a1a2214f8c8ec599c711514d58b984f15a6f32b493de3679ca26e7a"} Nov 02 06:27:12.872515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.872489 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerStarted","Data":"6e807a11475874d2bb97163c52f9e5c6c829381b0bf89e63bdde5bac855f5c17"} Nov 02 06:27:12.982112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.981973 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podStartSLOduration=1.981952449 podStartE2EDuration="1.981952449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:12.980408545 +0000 UTC m=+586.953743231" watchObservedRunningTime="2026-11-02 06:27:12.981952449 +0000 UTC m=+586.955287133" Nov 02 06:27:12.985025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.984980 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:27:12.989191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:12.989169 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.044240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.044207 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:27:13.071650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.071618 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-680b7b-openshift-default-dockercfg-sccx9\"" Nov 02 06:27:13.099556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.099416 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmg5\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.099556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.099463 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100386 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100444 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100476 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100518 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100549 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100594 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.100746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.100615 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.103527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.103500 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:27:13.105367 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:13.105344 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8f252c_df8b_4451_8934_3f753886c065.slice/crio-63276fa3a4f46d2e0a26f99d0be12f6c1293b50a41e5bf85f10fcb737e499b34 WatchSource:0}: Error finding container 63276fa3a4f46d2e0a26f99d0be12f6c1293b50a41e5bf85f10fcb737e499b34: Status 404 returned error can't find the container with id 63276fa3a4f46d2e0a26f99d0be12f6c1293b50a41e5bf85f10fcb737e499b34 Nov 02 06:27:13.107693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.107656 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.107752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.107735 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.107806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.107788 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202238 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202289 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202340 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202371 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202419 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202443 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202510 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmg5\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202542 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202665 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202831 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.202911 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.203338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.203300 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.204087 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.203414 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.204087 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.203955 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.205955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.205931 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.206957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.206931 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.259882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.259812 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.261864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.261836 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmg5\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.302094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.302067 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:13.694934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.694909 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:27:13.697562 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:13.697536 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e36a03_cf8c_487d_acea_5e9540769be8.slice/crio-9b419585f3648d7f7e83a3aa733caf21d2566077d5edfd19e7494903cc5489d7 WatchSource:0}: Error finding container 9b419585f3648d7f7e83a3aa733caf21d2566077d5edfd19e7494903cc5489d7: Status 404 returned error can't find the container with id 9b419585f3648d7f7e83a3aa733caf21d2566077d5edfd19e7494903cc5489d7 Nov 02 06:27:13.699811 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.699773 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.699929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.699847 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.699929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.699881 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.837055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.837012 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:13.841751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.841725 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:13.879202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.879161 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerStarted","Data":"d8ddcc2eb67662a37995dfcf957ee9e219498c8ec0c8fa7da54a600fa3d35832"} Nov 02 06:27:13.879385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.879223 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerStarted","Data":"63276fa3a4f46d2e0a26f99d0be12f6c1293b50a41e5bf85f10fcb737e499b34"} Nov 02 06:27:13.880914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.880888 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerStarted","Data":"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec"} Nov 02 06:27:13.882565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.882539 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerStarted","Data":"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f"} Nov 02 06:27:13.882680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.882585 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerStarted","Data":"9b419585f3648d7f7e83a3aa733caf21d2566077d5edfd19e7494903cc5489d7"} Nov 02 06:27:13.882739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.882728 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:13.883599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:13.883578 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:14.177548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.177493 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podStartSLOduration=2.177474939 podStartE2EDuration="2.177474939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:14.177246865 +0000 UTC m=+588.150581544" watchObservedRunningTime="2026-11-02 06:27:14.177474939 +0000 UTC m=+588.150809623" Nov 02 06:27:14.296909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.296839 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podStartSLOduration=2.296824849 podStartE2EDuration="2.296824849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:14.295069303 +0000 UTC m=+588.268403996" watchObservedRunningTime="2026-11-02 06:27:14.296824849 +0000 UTC m=+588.270159532" Nov 02 06:27:14.302942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.302916 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:14.307420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.307395 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:14.406646 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.406258 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podStartSLOduration=2.40624111 podStartE2EDuration="2.40624111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:14.405495355 +0000 UTC m=+588.378830038" watchObservedRunningTime="2026-11-02 06:27:14.40624111 +0000 UTC m=+588.379575795" Nov 02 06:27:14.438961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.438869 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:14.444348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.444325 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:14.724044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.723949 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:14.729549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.729526 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:14.887156 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.887122 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:14.887502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.887481 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:14.887717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.887704 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:14.888128 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.888046 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:14.888299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.888278 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:27:14.888450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:14.888435 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:27:30.673546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:30.673513 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:27:34.095678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.095643 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:27:34.100723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.100703 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.153305 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.153267 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-bdcd80\"" Nov 02 06:27:34.198800 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.198763 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-bdcd80-dockercfg-zr8pf\"" Nov 02 06:27:34.200281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.200252 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:27:34.224157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.224130 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.224275 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.224245 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gx4m\" (UniqueName: \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.261860 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.261831 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:27:34.265755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.265737 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.317393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.317369 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:27:34.324951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.324927 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx4m\" (UniqueName: \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.325073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.325010 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.325698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.325679 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.378759 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.378699 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx4m\" (UniqueName: \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\") pod \"payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.416585 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.416561 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:34.426138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.426022 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.426187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.426172 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rvh\" (UniqueName: \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.526839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.526800 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rvh\" (UniqueName: \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.527029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.526927 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.527614 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.527594 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.581744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.581711 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rvh\" (UniqueName: \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\") pod \"payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.782322 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.782300 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:27:34.785557 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:34.785526 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e99635_ca67_4650_a3fb_87605bf91888.slice/crio-9b2432f2bd1ba7c544d5f991906381528b3480605bad2a85ccb0644e3fcf2ac4 WatchSource:0}: Error finding container 9b2432f2bd1ba7c544d5f991906381528b3480605bad2a85ccb0644e3fcf2ac4: Status 404 returned error can't find the container with id 9b2432f2bd1ba7c544d5f991906381528b3480605bad2a85ccb0644e3fcf2ac4 Nov 02 06:27:34.878624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.878595 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:34.982837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:34.982806 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" event={"ID":"85e99635-ca67-4650-a3fb-87605bf91888","Type":"ContainerStarted","Data":"9b2432f2bd1ba7c544d5f991906381528b3480605bad2a85ccb0644e3fcf2ac4"} Nov 02 06:27:35.241455 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:35.241424 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:27:35.244878 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:35.244839 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c82080f_1376_447a_ae5d_dc9bf859db25.slice/crio-0d094ee166cb0edcda0a498ab5c1a4f2db350703a37bd2061c5d50549535d2a0 WatchSource:0}: Error finding container 0d094ee166cb0edcda0a498ab5c1a4f2db350703a37bd2061c5d50549535d2a0: Status 404 returned error can't find the container with id 0d094ee166cb0edcda0a498ab5c1a4f2db350703a37bd2061c5d50549535d2a0 Nov 02 06:27:35.990448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:35.990402 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" event={"ID":"0c82080f-1376-447a-ae5d-dc9bf859db25","Type":"ContainerStarted","Data":"0d094ee166cb0edcda0a498ab5c1a4f2db350703a37bd2061c5d50549535d2a0"} Nov 02 06:27:36.389343 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.389265 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:27:36.393782 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.393762 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.460504 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.460419 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:27:36.507337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.507299 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-fb1398-dockercfg-tj7bd\"" Nov 02 06:27:36.507521 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.507372 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-fb1398\"" Nov 02 06:27:36.556309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.556281 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:27:36.559270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.559236 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.559393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.559309 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmmb\" (UniqueName: \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.561084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.561062 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.613724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.613656 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:27:36.660601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.660570 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxts\" (UniqueName: \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.660768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.660678 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.660768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.660727 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.660768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.660756 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmmb\" (UniqueName: \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.661894 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.661866 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.716030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.715186 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmmb\" (UniqueName: \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\") pod \"payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:36.761549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.761521 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxts\" (UniqueName: \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.761687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.761609 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.762408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.762383 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.816580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.816555 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxts\" (UniqueName: \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\") pod \"payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:36.880385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:36.880092 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:37.012079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:37.012051 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:37.261873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:37.261797 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:27:37.400668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:37.400635 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:27:37.404683 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:37.404649 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c14fdbc_7eed_4120_8665_4d011bcd965d.slice/crio-e1569c0811119fa3745839c302295825c43cee61e3ba23de2f541d63a80f7461 WatchSource:0}: Error finding container e1569c0811119fa3745839c302295825c43cee61e3ba23de2f541d63a80f7461: Status 404 returned error can't find the container with id e1569c0811119fa3745839c302295825c43cee61e3ba23de2f541d63a80f7461 Nov 02 06:27:37.981779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:37.981746 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:27:38.006774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:38.006729 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" event={"ID":"4c14fdbc-7eed-4120-8665-4d011bcd965d","Type":"ContainerStarted","Data":"e1569c0811119fa3745839c302295825c43cee61e3ba23de2f541d63a80f7461"} Nov 02 06:27:38.008310 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:38.008277 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" event={"ID":"2fb43458-8056-4472-89a1-b8a06d3eeb3d","Type":"ContainerStarted","Data":"175a3bf0b9357195ab8ad03879ddc209232d2ca536a837cdea289952a14f8a87"} Nov 02 06:27:38.578983 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:27:38.578888 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:27:39.015347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.015312 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" event={"ID":"0c82080f-1376-447a-ae5d-dc9bf859db25","Type":"ContainerStarted","Data":"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75"} Nov 02 06:27:39.015570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.015483 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:39.016926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.016892 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" event={"ID":"4c14fdbc-7eed-4120-8665-4d011bcd965d","Type":"ContainerStarted","Data":"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234"} Nov 02 06:27:39.017088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.017005 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:39.018431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.018407 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" event={"ID":"2fb43458-8056-4472-89a1-b8a06d3eeb3d","Type":"ContainerStarted","Data":"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1"} Nov 02 06:27:39.018550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.018461 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:39.019850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.019828 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" event={"ID":"85e99635-ca67-4650-a3fb-87605bf91888","Type":"ContainerStarted","Data":"ad92f5e10e0059d57355b54ffb0e33f893c41f454428348068ce88548135454d"} Nov 02 06:27:39.019956 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.019882 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:39.126593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.126540 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" podStartSLOduration=2.170853463 podStartE2EDuration="5.126525278s" totalImagesPullingTime="2.955671815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:39.126229776 +0000 UTC m=+613.099564474" watchObservedRunningTime="2026-11-02 06:27:39.126525278 +0000 UTC m=+613.099860001" Nov 02 06:27:39.233561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.233495 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" podStartSLOduration=2.308877373 podStartE2EDuration="3.23347634s" totalImagesPullingTime="924.598967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:39.232332817 +0000 UTC m=+613.205667515" watchObservedRunningTime="2026-11-02 06:27:39.23347634 +0000 UTC m=+613.206811025" Nov 02 06:27:39.340064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.339930 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" podStartSLOduration=2.082083383 podStartE2EDuration="5.339916602s" totalImagesPullingTime="3.257833219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:39.338519443 +0000 UTC m=+613.311854128" watchObservedRunningTime="2026-11-02 06:27:39.339916602 +0000 UTC m=+613.313251286" Nov 02 06:27:39.465019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:39.456090 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" podStartSLOduration=2.549481462 podStartE2EDuration="3.456065412s" totalImagesPullingTime="906.58395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:39.452489535 +0000 UTC m=+613.425824221" watchObservedRunningTime="2026-11-02 06:27:39.456065412 +0000 UTC m=+613.429400096" Nov 02 06:27:42.186242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.186201 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:27:42.189978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.189957 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.235573 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.235536 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.235745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.235658 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p66d\" (UniqueName: \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.235868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.235840 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.236025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.235962 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.239244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.239225 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:27:42.285515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.285474 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-dcf9c3-serving-cert\"" Nov 02 06:27:42.285515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.285499 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:27:42.285760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.285534 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 06:27:42.285760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.285651 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 06:27:42.287311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.287291 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:27:42.336981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.336944 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.337190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.337065 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.337190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.337103 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.337190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.337173 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p66d\" (UniqueName: \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.337363 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:27:42.337228 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-serving-cert: secret "maas-api-e2e-worker-w5-dcf9c3-serving-cert" not found Nov 02 06:27:42.337363 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:27:42.337315 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls podName:f739d1f2-cf87-4e34-8070-b05b5d4d1563 nodeName:}" failed. No retries permitted until 2026-11-02 06:27:42.837292377 +0000 UTC m=+616.810627060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls") pod "maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" (UID: "f739d1f2-cf87-4e34-8070-b05b5d4d1563") : secret "maas-api-e2e-worker-w5-dcf9c3-serving-cert" not found Nov 02 06:27:42.338421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.338388 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.339874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.339847 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.392428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.392392 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p66d\" (UniqueName: \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.845085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.845041 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:42.890685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:42.890644 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:43.103241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:43.103157 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:43.497325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:43.497286 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:27:43.500304 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:27:43.500275 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf739d1f2_cf87_4e34_8070_b05b5d4d1563.slice/crio-c885cb7385930dbde0d1be1a54e2ea1519e99e427e069f0bb3130a152ff11297 WatchSource:0}: Error finding container c885cb7385930dbde0d1be1a54e2ea1519e99e427e069f0bb3130a152ff11297: Status 404 returned error can't find the container with id c885cb7385930dbde0d1be1a54e2ea1519e99e427e069f0bb3130a152ff11297 Nov 02 06:27:44.048933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:44.048833 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" event={"ID":"f739d1f2-cf87-4e34-8070-b05b5d4d1563","Type":"ContainerStarted","Data":"c885cb7385930dbde0d1be1a54e2ea1519e99e427e069f0bb3130a152ff11297"} Nov 02 06:27:45.931213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:45.931147 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:27:45.931686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:45.931284 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:27:48.090801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.090755 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" event={"ID":"f739d1f2-cf87-4e34-8070-b05b5d4d1563","Type":"ContainerStarted","Data":"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3"} Nov 02 06:27:48.091284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.090854 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:48.092321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.092298 6273 generic.go:356] "Generic (PLEG): container finished" podID="67e36a03-cf8c-487d-acea-5e9540769be8" containerID="6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f" exitCode=0 Nov 02 06:27:48.092405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.092331 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerDied","Data":"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f"} Nov 02 06:27:48.092612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.092600 6273 scope.go:122] "RemoveContainer" containerID="6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f" Nov 02 06:27:48.093798 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.093773 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:48.093848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.093835 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:48.093885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.093870 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:48.203658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:48.203471 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" podStartSLOduration=2.520974663 podStartE2EDuration="6.203451634s" totalImagesPullingTime="3.682476971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:48.201764716 +0000 UTC m=+622.175099412" watchObservedRunningTime="2026-11-02 06:27:48.203451634 +0000 UTC m=+622.176786319" Nov 02 06:27:49.105825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:49.105786 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerStarted","Data":"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0"} Nov 02 06:27:49.303013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:49.302478 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:49.303013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:49.302701 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 06:27:49.303013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:49.302741 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 06:27:50.028350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.028319 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:27:50.028560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.028409 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:27:50.028560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.028552 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:27:50.028703 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.028596 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:27:50.303069 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.302410 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 06:27:50.303069 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:50.302478 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 06:27:51.303516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:51.303478 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 06:27:51.303910 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:51.303532 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 06:27:52.303076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:52.302961 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 06:27:52.303076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:52.303054 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 06:27:53.302866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:53.302831 6273 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 06:27:53.303421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:53.302889 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 06:27:53.305193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:53.305159 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:27:54.112858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:54.112823 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:27:54.307704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:54.307668 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:54.308250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:54.308229 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:54.308980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:54.308960 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:27:56.147631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.147546 6273 generic.go:356] "Generic (PLEG): container finished" podID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerID="04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6" exitCode=0 Nov 02 06:27:56.147631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.147615 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerDied","Data":"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6"} Nov 02 06:27:56.148525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.148185 6273 scope.go:122] "RemoveContainer" containerID="04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6" Nov 02 06:27:56.149562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.149530 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:56.149637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.149603 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:56.149677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:56.149639 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:57.176424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:57.176389 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerStarted","Data":"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb"} Nov 02 06:27:57.837004 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:57.836961 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:57.838270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:57.838246 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:27:57.838381 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:57.838292 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 06:27:58.837401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:58.837365 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:27:58.837791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:58.837419 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 06:27:58.883542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:58.883507 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:27:59.837312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:59.837274 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:27:59.837472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:27:59.837329 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 06:28:00.837180 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:00.837143 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:28:00.837379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:00.837212 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 06:28:01.836913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:01.836865 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:28:01.837418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:01.836928 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 06:28:02.841379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:02.841346 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:28:02.842417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:02.842394 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:28:05.776147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.776065 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 06:28:05.779094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.779072 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:05.828083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.828053 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-6ff85e-serving-cert\"" Nov 02 06:28:05.876141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.876106 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 06:28:05.907566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.907522 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8454\" (UniqueName: \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:05.907783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.907601 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:05.907783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.907633 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:05.907783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:05.907761 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.008729 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.008694 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8454\" (UniqueName: \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.008896 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.008746 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.008896 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.008871 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.009047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.009031 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.009094 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:28:06.009066 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-serving-cert: secret "maas-api-e2e-worker-w6-6ff85e-serving-cert" not found Nov 02 06:28:06.009169 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:28:06.009156 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls podName:52310f90-2bf9-43e9-9c48-015c8fa145de nodeName:}" failed. No retries permitted until 2026-11-02 06:28:06.509130999 +0000 UTC m=+640.482465665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls") pod "maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" (UID: "52310f90-2bf9-43e9-9c48-015c8fa145de") : secret "maas-api-e2e-worker-w6-6ff85e-serving-cert" not found Nov 02 06:28:06.009495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.009477 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.011476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.011449 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.071952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.071826 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8454\" (UniqueName: \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.515510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.515460 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.518096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.518068 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:06.690602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:06.690575 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:07.074359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:07.074335 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 06:28:07.076764 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:28:07.076736 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52310f90_2bf9_43e9_9c48_015c8fa145de.slice/crio-1e1e6bff18914c044356d33183bd36e39f09701bba399537813dc5c9b59b859b WatchSource:0}: Error finding container 1e1e6bff18914c044356d33183bd36e39f09701bba399537813dc5c9b59b859b: Status 404 returned error can't find the container with id 1e1e6bff18914c044356d33183bd36e39f09701bba399537813dc5c9b59b859b Nov 02 06:28:07.232168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:07.232136 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" event={"ID":"52310f90-2bf9-43e9-9c48-015c8fa145de","Type":"ContainerStarted","Data":"1e1e6bff18914c044356d33183bd36e39f09701bba399537813dc5c9b59b859b"} Nov 02 06:28:08.239701 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:08.239666 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" event={"ID":"52310f90-2bf9-43e9-9c48-015c8fa145de","Type":"ContainerStarted","Data":"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3"} Nov 02 06:28:08.241484 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:08.239920 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:08.348734 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:08.348686 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" podStartSLOduration=2.529624234 podStartE2EDuration="3.348672196s" totalImagesPullingTime="819.047962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:08.348045182 +0000 UTC m=+642.321379883" watchObservedRunningTime="2026-11-02 06:28:08.348672196 +0000 UTC m=+642.322006880" Nov 02 06:28:08.562176 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:28:08.562082 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:28:11.256254 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.256225 6273 generic.go:356] "Generic (PLEG): container finished" podID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerID="e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec" exitCode=0 Nov 02 06:28:11.256620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.256300 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerDied","Data":"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec"} Nov 02 06:28:11.256721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.256706 6273 scope.go:122] "RemoveContainer" containerID="e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec" Nov 02 06:28:11.258108 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.258081 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:11.258181 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.258148 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:11.258220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:11.258181 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:12.288091 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:12.288053 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerStarted","Data":"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36"} Nov 02 06:28:12.438387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:12.438354 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:28:12.439438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:12.439406 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:12.439593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:12.439467 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:12.481014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:12.459072 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:28:13.438643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:13.438605 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:13.439133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:13.438670 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:14.293197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:14.292162 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 06:28:14.439144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:14.439102 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:14.439516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:14.439169 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:14.894761 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:14.892096 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:28:15.439393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:15.439351 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:15.439813 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:15.439423 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:16.438514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:16.438479 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:16.438697 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:16.438552 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:17.439012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:17.438954 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 06:28:17.439421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:17.439025 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 06:28:18.442679 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:18.442651 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:28:18.443771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:18.443751 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:28:22.351116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.351078 6273 generic.go:356] "Generic (PLEG): container finished" podID="cd8f252c-df8b-4451-8934-3f753886c065" containerID="d8ddcc2eb67662a37995dfcf957ee9e219498c8ec0c8fa7da54a600fa3d35832" exitCode=0 Nov 02 06:28:22.351546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.351151 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerDied","Data":"d8ddcc2eb67662a37995dfcf957ee9e219498c8ec0c8fa7da54a600fa3d35832"} Nov 02 06:28:22.351546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.351503 6273 scope.go:122] "RemoveContainer" containerID="d8ddcc2eb67662a37995dfcf957ee9e219498c8ec0c8fa7da54a600fa3d35832" Nov 02 06:28:22.352830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.352797 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:22.352942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.352892 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:22.353032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:22.352941 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:23.382270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:23.382223 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerStarted","Data":"8c3af2b431e6d0a138f802ac4659fb447d929da854bf4fcf2f46834177156014"} Nov 02 06:28:23.723789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:23.723753 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:28:23.724018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:23.723939 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 06:28:23.724098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:23.724017 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 06:28:24.724340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:24.724307 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 06:28:24.724724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:24.724358 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 06:28:25.723912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:25.723878 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 06:28:25.724109 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:25.723943 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 06:28:26.723744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:26.723709 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 06:28:26.724161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:26.723764 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 06:28:27.723648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:27.723611 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 06:28:27.723820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:27.723664 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 06:28:28.728315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:28.728285 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:28:28.728826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:28.728584 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:28:28.729292 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:28.729276 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:28:32.082918 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:32.082882 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:28:38.553184 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:28:38.553138 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:28:52.204059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:52.204024 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:28:53.011767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.011691 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:28:53.016860 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.016831 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.113673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.113642 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"openshift-service-ca.crt\"" Nov 02 06:28:53.113830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.113751 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"kube-root-ca.crt\"" Nov 02 06:28:53.114193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.114180 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"default-dockercfg-hc4f4\"" Nov 02 06:28:53.114241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.114191 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"simulator-w5-dcf9c3-kserve-self-signed-certs\"" Nov 02 06:28:53.115922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.115903 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:28:53.178439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178409 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.178599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178476 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.178599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178542 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.178599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178589 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr766\" (UniqueName: \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.178720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178702 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.178801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.178727 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.279857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.279784 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.279861 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.279880 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.279919 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr766\" (UniqueName: \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.280233 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.280278 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.280306 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.280534 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.280683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.280621 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.282236 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.282213 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.282476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.282458 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.334165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.334135 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr766\" (UniqueName: \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\") pod \"simulator-w5-dcf9c3-kserve-6457bc5566-jws8z\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.628932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.628848 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:53.993430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:53.993358 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:28:53.996651 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:28:53.996613 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba3ef96_6d95_4fa3_963c_d055e6243a80.slice/crio-c2b6d5cf05cc592a47462014e9d3ebc0bd8baef8d3dc0696a59514f3599ef3dc WatchSource:0}: Error finding container c2b6d5cf05cc592a47462014e9d3ebc0bd8baef8d3dc0696a59514f3599ef3dc: Status 404 returned error can't find the container with id c2b6d5cf05cc592a47462014e9d3ebc0bd8baef8d3dc0696a59514f3599ef3dc Nov 02 06:28:54.560389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:54.560354 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" event={"ID":"6ba3ef96-6d95-4fa3-963c-d055e6243a80","Type":"ContainerStarted","Data":"c2b6d5cf05cc592a47462014e9d3ebc0bd8baef8d3dc0696a59514f3599ef3dc"} Nov 02 06:28:56.572626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:56.572589 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" event={"ID":"6ba3ef96-6d95-4fa3-963c-d055e6243a80","Type":"ContainerStarted","Data":"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179"} Nov 02 06:28:56.629460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:56.629426 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:28:56.680623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:56.680575 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podStartSLOduration=3.17043753 podStartE2EDuration="4.680562523s" totalImagesPullingTime="1.510124993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:56.67953295 +0000 UTC m=+690.652867635" watchObservedRunningTime="2026-11-02 06:28:56.680562523 +0000 UTC m=+690.653897207" Nov 02 06:28:59.489582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.489541 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 06:28:59.493534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.493515 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.539965 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.539926 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.540149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.540040 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.540149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.540090 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.540149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.540121 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w5q\" (UniqueName: \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.540338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.540316 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.540431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.540405 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.591036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.591008 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"premium-w2-581608-kserve-self-signed-certs\"" Nov 02 06:28:59.591202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.591076 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"kube-root-ca.crt\"" Nov 02 06:28:59.591202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.591076 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"openshift-service-ca.crt\"" Nov 02 06:28:59.591202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.591152 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"default-dockercfg-k692v\"" Nov 02 06:28:59.592815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.592795 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 06:28:59.641554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641520 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.641722 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641567 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.641722 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641619 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.641722 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641660 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w5q\" (UniqueName: \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.641916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641741 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.641916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641798 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.642037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.641979 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.642096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.642051 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.642152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.642104 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.644062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.644036 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.644177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.644103 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.696337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.696311 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w5q\" (UniqueName: \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\") pod \"premium-w2-581608-kserve-c899d9d6-9lxz8\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:28:59.805566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:28:59.805492 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:29:00.180723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:00.180696 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 06:29:00.185399 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:29:00.185373 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db6bfd2_6e42_40ed_9fb3_bfa026d677c6.slice/crio-c66bc74478c5d39d06f691240f2602dabc4634c178854772b6730eeca898eceb WatchSource:0}: Error finding container c66bc74478c5d39d06f691240f2602dabc4634c178854772b6730eeca898eceb: Status 404 returned error can't find the container with id c66bc74478c5d39d06f691240f2602dabc4634c178854772b6730eeca898eceb Nov 02 06:29:00.594043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:00.593930 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" event={"ID":"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6","Type":"ContainerStarted","Data":"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6"} Nov 02 06:29:00.594043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:00.593981 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" event={"ID":"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6","Type":"ContainerStarted","Data":"c66bc74478c5d39d06f691240f2602dabc4634c178854772b6730eeca898eceb"} Nov 02 06:29:00.707186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:00.707138 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podStartSLOduration=1.7071238210000002 podStartE2EDuration="1.707123821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:00.705682538 +0000 UTC m=+694.679017222" watchObservedRunningTime="2026-11-02 06:29:00.707123821 +0000 UTC m=+694.680458504" Nov 02 06:29:00.806124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:00.806091 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:29:03.629142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:03.629110 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:29:03.641850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:03.641828 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:29:04.627051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:04.627023 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:29:07.438625 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:07.438594 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:29:08.553422 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:29:08.553387 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:29:09.806068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:09.806037 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:29:09.818556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:09.818524 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:29:10.663922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:10.663892 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:29:18.207920 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.207886 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:29:18.211814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.211792 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.310393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.310359 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"kube-root-ca.crt\"" Nov 02 06:29:18.310560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.310444 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"default-dockercfg-mf5zh\"" Nov 02 06:29:18.310707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.310690 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"openshift-service-ca.crt\"" Nov 02 06:29:18.310759 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.310744 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"simulator-w3-fb1398-kserve-self-signed-certs\"" Nov 02 06:29:18.312537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.312520 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:29:18.348379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348349 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.348553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348407 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.348553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348444 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.348676 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348601 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.348676 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348630 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.348785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.348673 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcvw\" (UniqueName: \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450154 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450197 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450240 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcvw\" (UniqueName: \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450303 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450342 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450372 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450711 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450743 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.450864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.450824 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.452558 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.452537 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.452811 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.452792 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.503975 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.503909 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcvw\" (UniqueName: \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\") pod \"simulator-w3-fb1398-kserve-668c65f95c-9qfpb\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.523756 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.523731 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:18.897836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:18.897757 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:29:18.905019 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:29:18.904975 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b81124_c0e7_40f0_9082_a4875e207882.slice/crio-086a1337b2778d1f02c9e8261c05a823d5e16aa6de3a9293ddd1054801e9bedc WatchSource:0}: Error finding container 086a1337b2778d1f02c9e8261c05a823d5e16aa6de3a9293ddd1054801e9bedc: Status 404 returned error can't find the container with id 086a1337b2778d1f02c9e8261c05a823d5e16aa6de3a9293ddd1054801e9bedc Nov 02 06:29:19.716921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:19.716884 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" event={"ID":"80b81124-c0e7-40f0-9082-a4875e207882","Type":"ContainerStarted","Data":"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8"} Nov 02 06:29:19.717489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:19.717463 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" event={"ID":"80b81124-c0e7-40f0-9082-a4875e207882","Type":"ContainerStarted","Data":"086a1337b2778d1f02c9e8261c05a823d5e16aa6de3a9293ddd1054801e9bedc"} Nov 02 06:29:19.825871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:19.825816 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podStartSLOduration=1.82580205 podStartE2EDuration="1.82580205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:19.824122507 +0000 UTC m=+713.797457190" watchObservedRunningTime="2026-11-02 06:29:19.82580205 +0000 UTC m=+713.799136734" Nov 02 06:29:20.524580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:20.524545 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:23.655051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:23.655017 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:29:28.524039 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:28.524000 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:28.536186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:28.536156 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:28.774621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:28.774544 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:29:38.553672 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:29:38.553590 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:29:51.099367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:29:51.099334 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:30:00.164213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.164174 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:00.173802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.173768 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:00.174580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.174552 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.179271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.179251 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.187865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.187840 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:00.191235 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.191213 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.223037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.223016 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:30:00.262263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.262217 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:00.270139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.270104 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:00.293026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.292969 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhf4\" (UniqueName: \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.293204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.293039 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.293204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.293091 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.293204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.293175 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.293440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.293239 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fcl\" (UniqueName: \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.293440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.293290 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.307733 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.307704 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:00.394755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394711 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhf4\" (UniqueName: \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.394755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394755 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.395055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394796 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.395055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394835 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.395055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394889 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fcl\" (UniqueName: \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.395055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.394924 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.395736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.395706 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.395736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.395724 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.395891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.395707 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.450868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.450796 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\") pod \"maas-api-key-cleanup-29893350-vqn82\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.451026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.450908 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhf4\" (UniqueName: \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.461479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.461448 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fcl\" (UniqueName: \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.493172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.493140 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:00.506384 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.506358 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:00.516008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.515970 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:00.884233 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.884138 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:00.892755 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:30:00.892730 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839c570e_7623_46ab_a747_0b6a6162285c.slice/crio-012d33624af6a8f9fecba8aa5d02effa50fdd22d5c12d512db43fb8c06d13fca WatchSource:0}: Error finding container 012d33624af6a8f9fecba8aa5d02effa50fdd22d5c12d512db43fb8c06d13fca: Status 404 returned error can't find the container with id 012d33624af6a8f9fecba8aa5d02effa50fdd22d5c12d512db43fb8c06d13fca Nov 02 06:30:00.934572 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.934541 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:00.936868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.936837 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:00.937634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:00.937609 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerStarted","Data":"012d33624af6a8f9fecba8aa5d02effa50fdd22d5c12d512db43fb8c06d13fca"} Nov 02 06:30:00.941315 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:30:00.941290 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9e660a_1064_400c_ae83_c70b621120f3.slice/crio-28c314cd4757ad725b485be553a394d6699ad39ed536d3ecaa823565a6f0784c WatchSource:0}: Error finding container 28c314cd4757ad725b485be553a394d6699ad39ed536d3ecaa823565a6f0784c: Status 404 returned error can't find the container with id 28c314cd4757ad725b485be553a394d6699ad39ed536d3ecaa823565a6f0784c Nov 02 06:30:00.943601 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:30:00.943578 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3424144_034c_4331_a946_6f3647fdb2bd.slice/crio-ac95471dc20acb354ecd7325c378181b4ee51908ae36fbf23055837af7b2bfff WatchSource:0}: Error finding container ac95471dc20acb354ecd7325c378181b4ee51908ae36fbf23055837af7b2bfff: Status 404 returned error can't find the container with id ac95471dc20acb354ecd7325c378181b4ee51908ae36fbf23055837af7b2bfff Nov 02 06:30:01.945573 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:01.945541 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerStarted","Data":"ac95471dc20acb354ecd7325c378181b4ee51908ae36fbf23055837af7b2bfff"} Nov 02 06:30:01.946827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:01.946800 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerStarted","Data":"28c314cd4757ad725b485be553a394d6699ad39ed536d3ecaa823565a6f0784c"} Nov 02 06:30:07.987548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.987519 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerStarted","Data":"48341838a4e8964ca0f5ab653d180a26eded9942c71c045c424a551b4a397345"} Nov 02 06:30:07.989271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.989247 6273 generic.go:356] "Generic (PLEG): container finished" podID="3a9e660a-1064-400c-ae83-c70b621120f3" containerID="bf56759c70a44bd356fb20f1069a5c16f08028891250c7055dd3bfc73472df27" exitCode=60 Nov 02 06:30:07.989368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.989322 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerDied","Data":"bf56759c70a44bd356fb20f1069a5c16f08028891250c7055dd3bfc73472df27"} Nov 02 06:30:07.989544 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.989526 6273 scope.go:122] "RemoveContainer" containerID="bf56759c70a44bd356fb20f1069a5c16f08028891250c7055dd3bfc73472df27" Nov 02 06:30:07.990959 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.990940 6273 generic.go:356] "Generic (PLEG): container finished" podID="b3424144-034c-4331-a946-6f3647fdb2bd" containerID="81ed24580de1f2385a5c5704db07747c3e85eaf5ba3ee37ce877642d0447e93a" exitCode=60 Nov 02 06:30:07.991065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.990978 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerDied","Data":"81ed24580de1f2385a5c5704db07747c3e85eaf5ba3ee37ce877642d0447e93a"} Nov 02 06:30:07.991270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:07.991252 6273 scope.go:122] "RemoveContainer" containerID="81ed24580de1f2385a5c5704db07747c3e85eaf5ba3ee37ce877642d0447e93a" Nov 02 06:30:08.098882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:08.098823 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" podStartSLOduration=1.693243942 podStartE2EDuration="8.09881007s" totalImagesPullingTime="6.405566128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:08.096572067 +0000 UTC m=+762.069906752" watchObservedRunningTime="2026-11-02 06:30:08.09881007 +0000 UTC m=+762.072144803" Nov 02 06:30:08.554711 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:08.554621 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:30:08.997918 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:08.997888 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerStarted","Data":"57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23"} Nov 02 06:30:08.999540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:08.999517 6273 generic.go:356] "Generic (PLEG): container finished" podID="839c570e-7623-46ab-a747-0b6a6162285c" containerID="48341838a4e8964ca0f5ab653d180a26eded9942c71c045c424a551b4a397345" exitCode=60 Nov 02 06:30:08.999674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:08.999595 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerDied","Data":"48341838a4e8964ca0f5ab653d180a26eded9942c71c045c424a551b4a397345"} Nov 02 06:30:08.999786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:08.999763 6273 scope.go:122] "RemoveContainer" containerID="48341838a4e8964ca0f5ab653d180a26eded9942c71c045c424a551b4a397345" Nov 02 06:30:09.002076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.002053 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerStarted","Data":"ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0"} Nov 02 06:30:09.108250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.108189 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" podStartSLOduration=3.002667685 podStartE2EDuration="9.108177135s" totalImagesPullingTime="6.10550945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:09.107690623 +0000 UTC m=+763.081025335" watchObservedRunningTime="2026-11-02 06:30:09.108177135 +0000 UTC m=+763.081511820" Nov 02 06:30:09.320610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.320558 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" podStartSLOduration=3.217224794 podStartE2EDuration="9.320542586s" totalImagesPullingTime="6.103317792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:09.319482188 +0000 UTC m=+763.292816873" watchObservedRunningTime="2026-11-02 06:30:09.320542586 +0000 UTC m=+763.293877270" Nov 02 06:30:09.717075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.717039 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:30:09.720768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.720747 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.770197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.770162 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"premium-w5-dcf9c3-kserve-self-signed-certs\"" Nov 02 06:30:09.775913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.775887 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:30:09.801202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801176 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.801332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801215 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.801387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801368 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.801429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801406 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.801466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801430 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lkk\" (UniqueName: \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.801517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.801502 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903103 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903146 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903172 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903234 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903264 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903286 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lkk\" (UniqueName: \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903571 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903545 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903622 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.903651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.903635 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.905419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.905397 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.905671 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.905653 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:09.957270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:09.957249 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lkk\" (UniqueName: \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\") pod \"premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:10.008927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.008837 6273 generic.go:356] "Generic (PLEG): container finished" podID="839c570e-7623-46ab-a747-0b6a6162285c" containerID="254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3" exitCode=60 Nov 02 06:30:10.009370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.008915 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerDied","Data":"254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3"} Nov 02 06:30:10.009370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.008976 6273 scope.go:122] "RemoveContainer" containerID="48341838a4e8964ca0f5ab653d180a26eded9942c71c045c424a551b4a397345" Nov 02 06:30:10.009370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.009251 6273 scope.go:122] "RemoveContainer" containerID="254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3" Nov 02 06:30:10.009572 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:10.009547 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d_odh-ai-gateway-infra(839c570e-7623-46ab-a747-0b6a6162285c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" podUID="839c570e-7623-46ab-a747-0b6a6162285c" Nov 02 06:30:10.011021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.010974 6273 generic.go:356] "Generic (PLEG): container finished" podID="3a9e660a-1064-400c-ae83-c70b621120f3" containerID="ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0" exitCode=60 Nov 02 06:30:10.011142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.011016 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerDied","Data":"ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0"} Nov 02 06:30:10.011281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.011266 6273 scope.go:122] "RemoveContainer" containerID="ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0" Nov 02 06:30:10.011481 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:10.011461 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7_odh-ai-gateway-infra(3a9e660a-1064-400c-ae83-c70b621120f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" Nov 02 06:30:10.012760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.012740 6273 generic.go:356] "Generic (PLEG): container finished" podID="b3424144-034c-4331-a946-6f3647fdb2bd" containerID="57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23" exitCode=60 Nov 02 06:30:10.012847 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.012768 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerDied","Data":"57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23"} Nov 02 06:30:10.012996 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.012971 6273 scope.go:122] "RemoveContainer" containerID="57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23" Nov 02 06:30:10.013179 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:10.013164 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893350-vqn82_odh-ai-gateway-infra(b3424144-034c-4331-a946-6f3647fdb2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" Nov 02 06:30:10.021933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.021911 6273 scope.go:122] "RemoveContainer" containerID="bf56759c70a44bd356fb20f1069a5c16f08028891250c7055dd3bfc73472df27" Nov 02 06:30:10.032901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.032880 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:10.034350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.034322 6273 scope.go:122] "RemoveContainer" containerID="81ed24580de1f2385a5c5704db07747c3e85eaf5ba3ee37ce877642d0447e93a" Nov 02 06:30:10.493439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:10.493407 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:30:10.500596 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:30:10.500567 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5380a83d_1c98_4d61_b2f8_dd7777741973.slice/crio-d2bb05bb3c86e730662b5c386a5fe108517df1d5cc77858800d5586e2fa1dd70 WatchSource:0}: Error finding container d2bb05bb3c86e730662b5c386a5fe108517df1d5cc77858800d5586e2fa1dd70: Status 404 returned error can't find the container with id d2bb05bb3c86e730662b5c386a5fe108517df1d5cc77858800d5586e2fa1dd70 Nov 02 06:30:11.020277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.020250 6273 scope.go:122] "RemoveContainer" containerID="ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0" Nov 02 06:30:11.020717 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:11.020486 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7_odh-ai-gateway-infra(3a9e660a-1064-400c-ae83-c70b621120f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" Nov 02 06:30:11.021851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.021835 6273 scope.go:122] "RemoveContainer" containerID="57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23" Nov 02 06:30:11.022086 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:11.022063 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893350-vqn82_odh-ai-gateway-infra(b3424144-034c-4331-a946-6f3647fdb2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" Nov 02 06:30:11.023126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.023105 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" event={"ID":"5380a83d-1c98-4d61-b2f8-dd7777741973","Type":"ContainerStarted","Data":"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec"} Nov 02 06:30:11.023224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.023139 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" event={"ID":"5380a83d-1c98-4d61-b2f8-dd7777741973","Type":"ContainerStarted","Data":"d2bb05bb3c86e730662b5c386a5fe108517df1d5cc77858800d5586e2fa1dd70"} Nov 02 06:30:11.024709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.024693 6273 scope.go:122] "RemoveContainer" containerID="254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3" Nov 02 06:30:11.024851 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:11.024835 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d_odh-ai-gateway-infra(839c570e-7623-46ab-a747-0b6a6162285c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" podUID="839c570e-7623-46ab-a747-0b6a6162285c" Nov 02 06:30:11.033462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.033447 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:11.233514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:11.233464 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podStartSLOduration=2.233450794 podStartE2EDuration="2.233450794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:11.232937641 +0000 UTC m=+765.206272336" watchObservedRunningTime="2026-11-02 06:30:11.233450794 +0000 UTC m=+765.206785477" Nov 02 06:30:12.450584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:12.450554 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:30:20.033363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:20.033335 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:20.045757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:20.045732 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:20.084892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:20.084868 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:30:20.173344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:20.173314 6273 scope.go:122] "RemoveContainer" containerID="57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23" Nov 02 06:30:21.081831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.081803 6273 generic.go:356] "Generic (PLEG): container finished" podID="b3424144-034c-4331-a946-6f3647fdb2bd" containerID="d550799332c97e5d29617e544566f4650c7dee24fcafe689a7030ce6d7a09042" exitCode=60 Nov 02 06:30:21.082264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.081878 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerDied","Data":"d550799332c97e5d29617e544566f4650c7dee24fcafe689a7030ce6d7a09042"} Nov 02 06:30:21.082264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.081942 6273 scope.go:122] "RemoveContainer" containerID="57debb073184b098cb8aca7dd48b4c794629071d0f96e4e50ed46ed2dfbe8f23" Nov 02 06:30:21.082385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.082303 6273 scope.go:122] "RemoveContainer" containerID="d550799332c97e5d29617e544566f4650c7dee24fcafe689a7030ce6d7a09042" Nov 02 06:30:21.082631 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:21.082593 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893350-vqn82_odh-ai-gateway-infra(b3424144-034c-4331-a946-6f3647fdb2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" Nov 02 06:30:21.172162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.172136 6273 scope.go:122] "RemoveContainer" containerID="254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3" Nov 02 06:30:21.172312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:21.172295 6273 scope.go:122] "RemoveContainer" containerID="ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0" Nov 02 06:30:22.089603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.089574 6273 generic.go:356] "Generic (PLEG): container finished" podID="3a9e660a-1064-400c-ae83-c70b621120f3" containerID="fa6b73e613f92406eb62552b62e36250e9b964e02bfa5a6fea68e0220bddaeda" exitCode=60 Nov 02 06:30:22.090038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.089640 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerDied","Data":"fa6b73e613f92406eb62552b62e36250e9b964e02bfa5a6fea68e0220bddaeda"} Nov 02 06:30:22.090038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.089708 6273 scope.go:122] "RemoveContainer" containerID="ab4a72945fb1a70f828b568b815da27b89a73ba05d178232a12f4727f03a42d0" Nov 02 06:30:22.090148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.090095 6273 scope.go:122] "RemoveContainer" containerID="fa6b73e613f92406eb62552b62e36250e9b964e02bfa5a6fea68e0220bddaeda" Nov 02 06:30:22.090538 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:22.090512 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7_odh-ai-gateway-infra(3a9e660a-1064-400c-ae83-c70b621120f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" Nov 02 06:30:22.093245 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.093226 6273 generic.go:356] "Generic (PLEG): container finished" podID="839c570e-7623-46ab-a747-0b6a6162285c" containerID="b7af112e0e57926569d3b31722f7ac4228f75d156e8a4660f8856fcc780c8de8" exitCode=60 Nov 02 06:30:22.093341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.093294 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerDied","Data":"b7af112e0e57926569d3b31722f7ac4228f75d156e8a4660f8856fcc780c8de8"} Nov 02 06:30:22.093598 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.093586 6273 scope.go:122] "RemoveContainer" containerID="b7af112e0e57926569d3b31722f7ac4228f75d156e8a4660f8856fcc780c8de8" Nov 02 06:30:22.093820 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:22.093802 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d_odh-ai-gateway-infra(839c570e-7623-46ab-a747-0b6a6162285c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" podUID="839c570e-7623-46ab-a747-0b6a6162285c" Nov 02 06:30:22.112658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.112635 6273 scope.go:122] "RemoveContainer" containerID="254a8cf18b322e2199378150440025de329df3b387267331f809f43f147a35a3" Nov 02 06:30:22.241160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.241072 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:22.399318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.399292 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:22.537881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.537805 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\") pod \"b3424144-034c-4331-a946-6f3647fdb2bd\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " Nov 02 06:30:22.538042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.537926 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\") pod \"b3424144-034c-4331-a946-6f3647fdb2bd\" (UID: \"b3424144-034c-4331-a946-6f3647fdb2bd\") " Nov 02 06:30:22.538178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.538154 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca" pod "b3424144-034c-4331-a946-6f3647fdb2bd" (UID: "b3424144-034c-4331-a946-6f3647fdb2bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:22.538398 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.538380 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3424144-034c-4331-a946-6f3647fdb2bd-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:22.540134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.540105 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx" pod "b3424144-034c-4331-a946-6f3647fdb2bd" (UID: "b3424144-034c-4331-a946-6f3647fdb2bd"). InnerVolumeSpecName "kube-api-access-bmmbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:22.639065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:22.639033 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/b3424144-034c-4331-a946-6f3647fdb2bd-kube-api-access-bmmbx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.102684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.102647 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" Nov 02 06:30:23.102684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.102663 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82" event={"ID":"b3424144-034c-4331-a946-6f3647fdb2bd","Type":"ContainerDied","Data":"ac95471dc20acb354ecd7325c378181b4ee51908ae36fbf23055837af7b2bfff"} Nov 02 06:30:23.103186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.102763 6273 scope.go:122] "RemoveContainer" containerID="d550799332c97e5d29617e544566f4650c7dee24fcafe689a7030ce6d7a09042" Nov 02 06:30:23.205954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.205911 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:23.257206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.257173 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:23.260969 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.260939 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893350-vqn82"] Nov 02 06:30:23.320741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.320714 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:23.379298 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.379224 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:23.448270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.448232 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\") pod \"3a9e660a-1064-400c-ae83-c70b621120f3\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " Nov 02 06:30:23.448430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.448392 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\" (UniqueName: \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\") pod \"3a9e660a-1064-400c-ae83-c70b621120f3\" (UID: \"3a9e660a-1064-400c-ae83-c70b621120f3\") " Nov 02 06:30:23.448541 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.448517 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca" pod "3a9e660a-1064-400c-ae83-c70b621120f3" (UID: "3a9e660a-1064-400c-ae83-c70b621120f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:23.448824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.448805 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a9e660a-1064-400c-ae83-c70b621120f3-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.450906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.450882 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4" pod "3a9e660a-1064-400c-ae83-c70b621120f3" (UID: "3a9e660a-1064-400c-ae83-c70b621120f3"). InnerVolumeSpecName "kube-api-access-kwhf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:23.495349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.495329 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:23.549997 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.549960 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhf4\" (UniqueName: \"kubernetes.io/projected/3a9e660a-1064-400c-ae83-c70b621120f3-kube-api-access-kwhf4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.650845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.650823 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\" (UniqueName: \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\") pod \"839c570e-7623-46ab-a747-0b6a6162285c\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " Nov 02 06:30:23.651018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.650886 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\") pod \"839c570e-7623-46ab-a747-0b6a6162285c\" (UID: \"839c570e-7623-46ab-a747-0b6a6162285c\") " Nov 02 06:30:23.651275 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.651249 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca" pod "839c570e-7623-46ab-a747-0b6a6162285c" (UID: "839c570e-7623-46ab-a747-0b6a6162285c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:23.652947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.652928 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl" pod "839c570e-7623-46ab-a747-0b6a6162285c" (UID: "839c570e-7623-46ab-a747-0b6a6162285c"). InnerVolumeSpecName "kube-api-access-n5fcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:23.752533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.752506 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fcl\" (UniqueName: \"kubernetes.io/projected/839c570e-7623-46ab-a747-0b6a6162285c-kube-api-access-n5fcl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.752533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:23.752530 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839c570e-7623-46ab-a747-0b6a6162285c-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:24.112677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.112596 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" Nov 02 06:30:24.113250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.112603 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7" event={"ID":"3a9e660a-1064-400c-ae83-c70b621120f3","Type":"ContainerDied","Data":"28c314cd4757ad725b485be553a394d6699ad39ed536d3ecaa823565a6f0784c"} Nov 02 06:30:24.113250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.112872 6273 scope.go:122] "RemoveContainer" containerID="fa6b73e613f92406eb62552b62e36250e9b964e02bfa5a6fea68e0220bddaeda" Nov 02 06:30:24.115033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.115005 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" event={"ID":"839c570e-7623-46ab-a747-0b6a6162285c","Type":"ContainerDied","Data":"012d33624af6a8f9fecba8aa5d02effa50fdd22d5c12d512db43fb8c06d13fca"} Nov 02 06:30:24.115203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.115103 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d" Nov 02 06:30:24.123928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.123902 6273 scope.go:122] "RemoveContainer" containerID="b7af112e0e57926569d3b31722f7ac4228f75d156e8a4660f8856fcc780c8de8" Nov 02 06:30:24.179378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.179347 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" path="/var/lib/kubelet/pods/b3424144-034c-4331-a946-6f3647fdb2bd/volumes" Nov 02 06:30:24.268746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.268704 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:24.271726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.271701 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893350-sj4f7"] Nov 02 06:30:24.373586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.373480 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:24.376300 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:24.376273 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893350-pgl2d"] Nov 02 06:30:26.181870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:26.180676 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" path="/var/lib/kubelet/pods/3a9e660a-1064-400c-ae83-c70b621120f3/volumes" Nov 02 06:30:26.181870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:26.181306 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839c570e-7623-46ab-a747-0b6a6162285c" path="/var/lib/kubelet/pods/839c570e-7623-46ab-a747-0b6a6162285c/volumes" Nov 02 06:30:32.214508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:32.214474 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:30:36.290007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.289962 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:30:36.290471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290457 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290473 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290483 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290488 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290505 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290510 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290518 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290523 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290529 6273 cpu_manager.go:414] "containerMap: removing container" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290533 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290546 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290550 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290558 6273 cpu_manager.go:414] "containerMap: removing container" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290562 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290637 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290643 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290652 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290659 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3424144-034c-4331-a946-6f3647fdb2bd" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290665 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290670 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290676 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:30:36.290687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.290681 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:30:36.295602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.295583 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.388207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.388169 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"kube-root-ca.crt\"" Nov 02 06:30:36.391191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.391167 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"premium-w0-bdcd80-kserve-self-signed-certs\"" Nov 02 06:30:36.391191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.391194 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"default-dockercfg-6lvdv\"" Nov 02 06:30:36.391359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.391197 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"openshift-service-ca.crt\"" Nov 02 06:30:36.393131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.393115 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:30:36.483794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.483762 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.483971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.483801 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.483971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.483827 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.483971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.483879 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dx8\" (UniqueName: \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.483971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.483952 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.484129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.484022 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.584721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.584631 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.584879 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.584718 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.584961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.584884 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585046 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.584962 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585046 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.585025 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.585048 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dx8\" (UniqueName: \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.585229 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.585347 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.585597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.585562 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.587290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.587272 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.587489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.587458 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.663026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.662972 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dx8\" (UniqueName: \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\") pod \"premium-w0-bdcd80-kserve-79866d64bf-pfzlg\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:36.907409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:36.907373 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:37.285346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:37.285268 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:30:37.292569 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:30:37.292541 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cfa8c44_a6b4_40ee_aaf1_8950f4762b62.slice/crio-1e48f02af00e31b4c9f90e2d7d1431ced7b50f36e433ef244ef1cdf704363b89 WatchSource:0}: Error finding container 1e48f02af00e31b4c9f90e2d7d1431ced7b50f36e433ef244ef1cdf704363b89: Status 404 returned error can't find the container with id 1e48f02af00e31b4c9f90e2d7d1431ced7b50f36e433ef244ef1cdf704363b89 Nov 02 06:30:38.191333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:38.191296 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" event={"ID":"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62","Type":"ContainerStarted","Data":"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c"} Nov 02 06:30:38.191333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:38.191339 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" event={"ID":"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62","Type":"ContainerStarted","Data":"1e48f02af00e31b4c9f90e2d7d1431ced7b50f36e433ef244ef1cdf704363b89"} Nov 02 06:30:38.301448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:38.301380 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podStartSLOduration=2.301366087 podStartE2EDuration="2.301366087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:38.29927821 +0000 UTC m=+792.272612893" watchObservedRunningTime="2026-11-02 06:30:38.301366087 +0000 UTC m=+792.274700765" Nov 02 06:30:38.552270 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:30:38.552172 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:30:38.907938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:38.907911 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:46.908382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:46.908347 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:46.920779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:46.920748 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:30:47.252830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:30:47.252745 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:31:08.552174 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:31:08.552087 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:31:10.829302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:10.829268 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:31:31.938984 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.938949 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84"] Nov 02 06:31:31.939655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.939599 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:31:31.939655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.939620 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:31:31.939655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.939641 6273 cpu_manager.go:414] "containerMap: removing container" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:31:31.939655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.939649 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c570e-7623-46ab-a747-0b6a6162285c" containerName="cleanup" Nov 02 06:31:31.939876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.939809 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9e660a-1064-400c-ae83-c70b621120f3" containerName="cleanup" Nov 02 06:31:31.943470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.943451 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:31.997131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:31.997106 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84"] Nov 02 06:31:32.040483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.040457 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:31:32.040615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.040504 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:31:32.110475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.110432 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.110617 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.110503 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.110617 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.110603 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjvc\" (UniqueName: \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.211263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.211175 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.211263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.211236 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.211504 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.211316 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjvc\" (UniqueName: \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.211850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.211827 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.213620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.213595 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.265077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.265054 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjvc\" (UniqueName: \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\") pod \"maas-api-revoke-sub-edf28a2f-znp84\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.556368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.556289 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:32.928464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:32.928430 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84"] Nov 02 06:31:32.935905 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:31:32.935877 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fbf207_0e2d_4ec9_9b58_6df8998dce2c.slice/crio-d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0 WatchSource:0}: Error finding container d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0: Status 404 returned error can't find the container with id d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0 Nov 02 06:31:33.509782 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:33.509701 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" event={"ID":"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c","Type":"ContainerStarted","Data":"2fdd848eeaf24c70c343fa3f5a313de3fb9fe4631cdf68c47db93a74496987f8"} Nov 02 06:31:33.509782 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:33.509755 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" event={"ID":"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c","Type":"ContainerStarted","Data":"d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0"} Nov 02 06:31:33.616769 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:33.616719 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" podStartSLOduration=2.6167073260000002 podStartE2EDuration="2.616707326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:33.615751542 +0000 UTC m=+847.589086243" watchObservedRunningTime="2026-11-02 06:31:33.616707326 +0000 UTC m=+847.590042000" Nov 02 06:31:34.516485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:34.516449 6273 generic.go:356] "Generic (PLEG): container finished" podID="f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" containerID="2fdd848eeaf24c70c343fa3f5a313de3fb9fe4631cdf68c47db93a74496987f8" exitCode=0 Nov 02 06:31:34.516876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:34.516533 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" event={"ID":"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c","Type":"ContainerDied","Data":"2fdd848eeaf24c70c343fa3f5a313de3fb9fe4631cdf68c47db93a74496987f8"} Nov 02 06:31:35.673759 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.673734 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:35.750836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.750811 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\") pod \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " Nov 02 06:31:35.751026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.750857 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\") pod \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " Nov 02 06:31:35.751026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.750885 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\" (UniqueName: \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\") pod \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\" (UID: \"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c\") " Nov 02 06:31:35.751247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.751221 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca" pod "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" (UID: "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:35.751435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.751417 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:35.752906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.752882 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca" pod "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" (UID: "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:31:35.752980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.752884 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc" pod "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" (UID: "f9fbf207-0e2d-4ec9-9b58-6df8998dce2c"). InnerVolumeSpecName "kube-api-access-9xjvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:35.852640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.852550 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:35.852640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:35.852587 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjvc\" (UniqueName: \"kubernetes.io/projected/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c-kube-api-access-9xjvc\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:36.547065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:36.543174 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" event={"ID":"f9fbf207-0e2d-4ec9-9b58-6df8998dce2c","Type":"ContainerDied","Data":"d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0"} Nov 02 06:31:36.547065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:36.543239 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d385fa6162bfcb93be2ce41590971aee057af01e268f251723d8e201ec6250f0" Nov 02 06:31:36.547065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:36.543306 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84" Nov 02 06:31:38.552126 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:31:38.552088 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:31:48.294901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:48.294867 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:31:48.295417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:48.295087 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" containerID="cri-o://62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454" gracePeriod=30 Nov 02 06:31:49.292578 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:49.292542 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:31:49.292752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:49.292596 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:31:53.562686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.562621 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:31:53.647302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.647267 6273 generic.go:356] "Generic (PLEG): container finished" podID="96141757-259e-4bbd-8615-884a8edd04af" containerID="62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454" exitCode=0 Nov 02 06:31:53.647485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.647323 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" Nov 02 06:31:53.647485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.647360 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" event={"ID":"96141757-259e-4bbd-8615-884a8edd04af","Type":"ContainerDied","Data":"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454"} Nov 02 06:31:53.647485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.647431 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8" event={"ID":"96141757-259e-4bbd-8615-884a8edd04af","Type":"ContainerDied","Data":"50f03679ad098825fedcc57283b98e985cce85dc8c4726bc92f3be564899aa01"} Nov 02 06:31:53.647485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.647466 6273 scope.go:122] "RemoveContainer" containerID="62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454" Nov 02 06:31:53.657376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.657355 6273 scope.go:122] "RemoveContainer" containerID="62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454" Nov 02 06:31:53.657655 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:31:53.657636 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454\": container with ID starting with 62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454 not found: ID does not exist" containerID="62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454" Nov 02 06:31:53.657724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.657668 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454"} err="failed to get container status \"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454\": rpc error: code = NotFound desc = could not find container \"62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454\": container with ID starting with 62eb66599eeae310894ab24b3069006592e85cc229509b494a426dfdfc42b454 not found: ID does not exist" Nov 02 06:31:53.669949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.669926 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670022 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670064 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670130 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670175 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670220 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670273 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670301 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670360 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:31:53.670413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670370 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\") pod \"96141757-259e-4bbd-8615-884a8edd04af\" (UID: \"96141757-259e-4bbd-8615-884a8edd04af\") " Nov 02 06:31:53.670413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670377 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670728 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670776 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670828 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670888 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96141757-259e-4bbd-8615-884a8edd04af-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670907 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670922 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.670966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.670937 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.672601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.672538 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:53.672705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.672603 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:31:53.672705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.672611 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "kube-api-access-8zt2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:53.672903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.672887 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo" pod "96141757-259e-4bbd-8615-884a8edd04af" (UID: "96141757-259e-4bbd-8615-884a8edd04af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:31:53.771814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.771785 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.771814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.771812 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96141757-259e-4bbd-8615-884a8edd04af-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.772018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.771822 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96141757-259e-4bbd-8615-884a8edd04af-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.772018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.771832 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zt2n\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-kube-api-access-8zt2n\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:53.772018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:53.771841 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96141757-259e-4bbd-8615-884a8edd04af-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:54.066814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:54.066779 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:31:54.069099 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:54.069071 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fnwg8"] Nov 02 06:31:54.180427 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:31:54.180397 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96141757-259e-4bbd-8615-884a8edd04af" path="/var/lib/kubelet/pods/96141757-259e-4bbd-8615-884a8edd04af/volumes" Nov 02 06:32:02.432859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.432819 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k"] Nov 02 06:32:02.433453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433436 6273 cpu_manager.go:414] "containerMap: removing container" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" Nov 02 06:32:02.433535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433453 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" Nov 02 06:32:02.433535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433474 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" containerName="revoke-keys" Nov 02 06:32:02.433535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433480 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" containerName="revoke-keys" Nov 02 06:32:02.433634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433568 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" containerName="revoke-keys" Nov 02 06:32:02.433634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.433584 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="96141757-259e-4bbd-8615-884a8edd04af" containerName="istio-proxy" Nov 02 06:32:02.438466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.438445 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.459172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.459147 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.459308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.459292 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfbd\" (UniqueName: \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.459376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.459335 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.535684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.535651 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:02.535878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.535797 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:02.537760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.537733 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k"] Nov 02 06:32:02.560307 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.560272 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfbd\" (UniqueName: \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.560458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.560313 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.560458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.560361 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.561006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.560972 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.562839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.562816 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.615365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.615337 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfbd\" (UniqueName: \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\") pod \"maas-api-revoke-sub-7fad2add-bfz5k\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:02.751367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:02.751289 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:03.120633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:03.120545 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k"] Nov 02 06:32:03.127641 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:03.127614 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358463fc_e677_463e_9a1f_9d2a48ad96a9.slice/crio-902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa WatchSource:0}: Error finding container 902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa: Status 404 returned error can't find the container with id 902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa Nov 02 06:32:03.702515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:03.702479 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" event={"ID":"358463fc-e677-463e-9a1f-9d2a48ad96a9","Type":"ContainerStarted","Data":"3d2d9d5ffca2016d4b976d66d3121ab0d035bccaf323691a773cbccf49fc6c41"} Nov 02 06:32:03.702890 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:03.702536 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" event={"ID":"358463fc-e677-463e-9a1f-9d2a48ad96a9","Type":"ContainerStarted","Data":"902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa"} Nov 02 06:32:03.809599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:03.809491 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" podStartSLOduration=1.809476898 podStartE2EDuration="1.809476898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:03.808544329 +0000 UTC m=+877.781879013" watchObservedRunningTime="2026-11-02 06:32:03.809476898 +0000 UTC m=+877.782811581" Nov 02 06:32:04.709324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:04.709289 6273 generic.go:356] "Generic (PLEG): container finished" podID="358463fc-e677-463e-9a1f-9d2a48ad96a9" containerID="3d2d9d5ffca2016d4b976d66d3121ab0d035bccaf323691a773cbccf49fc6c41" exitCode=0 Nov 02 06:32:04.709717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:04.709374 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" event={"ID":"358463fc-e677-463e-9a1f-9d2a48ad96a9","Type":"ContainerDied","Data":"3d2d9d5ffca2016d4b976d66d3121ab0d035bccaf323691a773cbccf49fc6c41"} Nov 02 06:32:05.858502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.858478 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:05.891745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.891725 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\" (UniqueName: \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\") pod \"358463fc-e677-463e-9a1f-9d2a48ad96a9\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " Nov 02 06:32:05.891831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.891783 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\") pod \"358463fc-e677-463e-9a1f-9d2a48ad96a9\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " Nov 02 06:32:05.891914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.891900 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\") pod \"358463fc-e677-463e-9a1f-9d2a48ad96a9\" (UID: \"358463fc-e677-463e-9a1f-9d2a48ad96a9\") " Nov 02 06:32:05.892147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.892120 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca" pod "358463fc-e677-463e-9a1f-9d2a48ad96a9" (UID: "358463fc-e677-463e-9a1f-9d2a48ad96a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:05.892367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.892350 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:05.893687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.893662 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd" pod "358463fc-e677-463e-9a1f-9d2a48ad96a9" (UID: "358463fc-e677-463e-9a1f-9d2a48ad96a9"). InnerVolumeSpecName "kube-api-access-vwfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:05.893837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.893818 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca" pod "358463fc-e677-463e-9a1f-9d2a48ad96a9" (UID: "358463fc-e677-463e-9a1f-9d2a48ad96a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:05.993792 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.993723 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358463fc-e677-463e-9a1f-9d2a48ad96a9-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:05.993792 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:05.993755 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwfbd\" (UniqueName: \"kubernetes.io/projected/358463fc-e677-463e-9a1f-9d2a48ad96a9-kube-api-access-vwfbd\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:06.676893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:06.676857 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:32:06.724345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:06.724319 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" Nov 02 06:32:06.724525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:06.724343 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k" event={"ID":"358463fc-e677-463e-9a1f-9d2a48ad96a9","Type":"ContainerDied","Data":"902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa"} Nov 02 06:32:06.724525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:06.724400 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902885138d8bcd4afbeb19f64f70ed3f2fe35b5a721b8c5119299b907033d0aa" Nov 02 06:32:08.553680 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:32:08.553642 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:32:18.228395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.228359 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq"] Nov 02 06:32:18.228882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.228867 6273 cpu_manager.go:414] "containerMap: removing container" podUID="358463fc-e677-463e-9a1f-9d2a48ad96a9" containerName="revoke-keys" Nov 02 06:32:18.228882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.228882 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358463fc-e677-463e-9a1f-9d2a48ad96a9" containerName="revoke-keys" Nov 02 06:32:18.228979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.228970 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="358463fc-e677-463e-9a1f-9d2a48ad96a9" containerName="revoke-keys" Nov 02 06:32:18.232265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.232242 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.285942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.285908 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq"] Nov 02 06:32:18.329386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.329340 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.329604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.329397 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9h9d\" (UniqueName: \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.329604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.329528 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.329745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.329712 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:18.329804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.329782 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:18.430428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.430392 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.430428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.430431 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9h9d\" (UniqueName: \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.430665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.430473 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.431171 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.431149 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.432733 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.432712 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.484201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.484135 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9h9d\" (UniqueName: \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\") pod \"maas-api-revoke-sub-6ade36c2-28tmq\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.545379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.545348 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:18.923875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:18.923846 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq"] Nov 02 06:32:18.931723 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:18.931699 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64ff033_ac80_40e9_a258_540c3c923d6d.slice/crio-f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3 WatchSource:0}: Error finding container f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3: Status 404 returned error can't find the container with id f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3 Nov 02 06:32:19.799197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:19.799158 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" event={"ID":"e64ff033-ac80-40e9-a258-540c3c923d6d","Type":"ContainerStarted","Data":"2fdec35bb705ff23e5c70578d8e0e0c75e9e1ea9393a098767a4e283bb75202c"} Nov 02 06:32:19.799590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:19.799250 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" event={"ID":"e64ff033-ac80-40e9-a258-540c3c923d6d","Type":"ContainerStarted","Data":"f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3"} Nov 02 06:32:19.908734 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:19.908667 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" podStartSLOduration=1.9086507639999999 podStartE2EDuration="1.908650764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:19.90613122 +0000 UTC m=+893.879465905" watchObservedRunningTime="2026-11-02 06:32:19.908650764 +0000 UTC m=+893.881985449" Nov 02 06:32:20.806565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:20.806527 6273 generic.go:356] "Generic (PLEG): container finished" podID="e64ff033-ac80-40e9-a258-540c3c923d6d" containerID="2fdec35bb705ff23e5c70578d8e0e0c75e9e1ea9393a098767a4e283bb75202c" exitCode=0 Nov 02 06:32:20.806954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:20.806617 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" event={"ID":"e64ff033-ac80-40e9-a258-540c3c923d6d","Type":"ContainerDied","Data":"2fdec35bb705ff23e5c70578d8e0e0c75e9e1ea9393a098767a4e283bb75202c"} Nov 02 06:32:21.952112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.952085 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:21.989424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.989393 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\") pod \"e64ff033-ac80-40e9-a258-540c3c923d6d\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " Nov 02 06:32:21.989552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.989438 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\") pod \"e64ff033-ac80-40e9-a258-540c3c923d6d\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " Nov 02 06:32:21.989552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.989484 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\" (UniqueName: \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\") pod \"e64ff033-ac80-40e9-a258-540c3c923d6d\" (UID: \"e64ff033-ac80-40e9-a258-540c3c923d6d\") " Nov 02 06:32:21.989764 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.989738 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca" pod "e64ff033-ac80-40e9-a258-540c3c923d6d" (UID: "e64ff033-ac80-40e9-a258-540c3c923d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:21.989863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.989847 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:21.991378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.991351 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca" pod "e64ff033-ac80-40e9-a258-540c3c923d6d" (UID: "e64ff033-ac80-40e9-a258-540c3c923d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:21.991546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:21.991522 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d" pod "e64ff033-ac80-40e9-a258-540c3c923d6d" (UID: "e64ff033-ac80-40e9-a258-540c3c923d6d"). InnerVolumeSpecName "kube-api-access-c9h9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:22.090771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.090694 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64ff033-ac80-40e9-a258-540c3c923d6d-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:22.090771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.090722 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9h9d\" (UniqueName: \"kubernetes.io/projected/e64ff033-ac80-40e9-a258-540c3c923d6d-kube-api-access-c9h9d\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:22.574851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.574812 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 06:32:22.575406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.575388 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e64ff033-ac80-40e9-a258-540c3c923d6d" containerName="revoke-keys" Nov 02 06:32:22.575453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.575409 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64ff033-ac80-40e9-a258-540c3c923d6d" containerName="revoke-keys" Nov 02 06:32:22.575525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.575507 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64ff033-ac80-40e9-a258-540c3c923d6d" containerName="revoke-keys" Nov 02 06:32:22.578753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.578731 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597155 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597185 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597424 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597468 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cz9\" (UniqueName: \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597614 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.597692 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.597674 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.630693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.630669 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"openshift-service-ca.crt\"" Nov 02 06:32:22.631019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.630983 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"kube-root-ca.crt\"" Nov 02 06:32:22.631121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.631038 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"default-dockercfg-8vqbx\"" Nov 02 06:32:22.672505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.672479 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"distinct-w6-6ff85e-kserve-self-signed-certs\"" Nov 02 06:32:22.679480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.679461 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 06:32:22.699060 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699036 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699073 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cz9\" (UniqueName: \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699117 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699162 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699193 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699218 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699540 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699549 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.699775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.699753 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.702151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.702124 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.702561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.702516 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.759667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.759632 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cz9\" (UniqueName: \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\") pod \"distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:22.819479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.819450 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" Nov 02 06:32:22.819479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.819476 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq" event={"ID":"e64ff033-ac80-40e9-a258-540c3c923d6d","Type":"ContainerDied","Data":"f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3"} Nov 02 06:32:22.819693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.819530 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cf29c528d95e8bdcf2e3bb68c3d29184c7c614a0569bec0bc4d14fccbb46c3" Nov 02 06:32:22.891824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:22.891745 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:23.283098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:23.283061 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 06:32:23.289813 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:23.289788 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17ad7deb_e433_4587_a419_cec6305034fc.slice/crio-9133b0ecf0b982f3dcfd493564cfba4cce8ebd2058afbc59ddf33fb0f5451c91 WatchSource:0}: Error finding container 9133b0ecf0b982f3dcfd493564cfba4cce8ebd2058afbc59ddf33fb0f5451c91: Status 404 returned error can't find the container with id 9133b0ecf0b982f3dcfd493564cfba4cce8ebd2058afbc59ddf33fb0f5451c91 Nov 02 06:32:23.826963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:23.826920 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" event={"ID":"17ad7deb-e433-4587-a419-cec6305034fc","Type":"ContainerStarted","Data":"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0"} Nov 02 06:32:23.827139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:23.826972 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" event={"ID":"17ad7deb-e433-4587-a419-cec6305034fc","Type":"ContainerStarted","Data":"9133b0ecf0b982f3dcfd493564cfba4cce8ebd2058afbc59ddf33fb0f5451c91"} Nov 02 06:32:23.891887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:23.891851 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:23.936134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:23.936081 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podStartSLOduration=1.936068387 podStartE2EDuration="1.936068387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:23.934614836 +0000 UTC m=+897.907949520" watchObservedRunningTime="2026-11-02 06:32:23.936068387 +0000 UTC m=+897.909403090" Nov 02 06:32:32.892306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:32.892272 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:32.904803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:32.904777 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:33.893209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:33.893134 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:32:34.311565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.311531 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm"] Nov 02 06:32:34.316471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.316448 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.366048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.366021 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:34.368387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.368356 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm"] Nov 02 06:32:34.368850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.368819 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:34.428503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.428478 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qh2z\" (UniqueName: \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.428638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.428513 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.428638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.428562 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.529648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.529617 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qh2z\" (UniqueName: \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.529804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.529652 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.529804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.529780 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.530256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.530234 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.532056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.532037 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.583635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.583561 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qh2z\" (UniqueName: \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\") pod \"maas-api-revoke-sub-992efb99-prxtm\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:34.629666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:34.629632 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:35.004262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:35.004229 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm"] Nov 02 06:32:35.011522 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:35.011493 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafebcaf_aae2_43da_aac0_990cd46c9ecb.slice/crio-b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b WatchSource:0}: Error finding container b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b: Status 404 returned error can't find the container with id b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b Nov 02 06:32:35.902352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:35.902318 6273 generic.go:356] "Generic (PLEG): container finished" podID="fafebcaf-aae2-43da-aac0-990cd46c9ecb" containerID="003c9cd66d0c3302f6baebebff98454c3e15a5588aa6f24ff4d0b8dcabaa104b" exitCode=0 Nov 02 06:32:35.902501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:35.902360 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" event={"ID":"fafebcaf-aae2-43da-aac0-990cd46c9ecb","Type":"ContainerDied","Data":"003c9cd66d0c3302f6baebebff98454c3e15a5588aa6f24ff4d0b8dcabaa104b"} Nov 02 06:32:35.902501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:35.902427 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" event={"ID":"fafebcaf-aae2-43da-aac0-990cd46c9ecb","Type":"ContainerStarted","Data":"b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b"} Nov 02 06:32:37.052202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.052176 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:37.157648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.157617 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\") pod \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " Nov 02 06:32:37.157898 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.157797 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\" (UniqueName: \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\") pod \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " Nov 02 06:32:37.158072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.158051 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\") pod \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\" (UID: \"fafebcaf-aae2-43da-aac0-990cd46c9ecb\") " Nov 02 06:32:37.158467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.158439 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca" pod "fafebcaf-aae2-43da-aac0-990cd46c9ecb" (UID: "fafebcaf-aae2-43da-aac0-990cd46c9ecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:37.160753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.160714 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca" pod "fafebcaf-aae2-43da-aac0-990cd46c9ecb" (UID: "fafebcaf-aae2-43da-aac0-990cd46c9ecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:37.163596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.163569 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z" pod "fafebcaf-aae2-43da-aac0-990cd46c9ecb" (UID: "fafebcaf-aae2-43da-aac0-990cd46c9ecb"). InnerVolumeSpecName "kube-api-access-5qh2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:37.259805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.259775 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qh2z\" (UniqueName: \"kubernetes.io/projected/fafebcaf-aae2-43da-aac0-990cd46c9ecb-kube-api-access-5qh2z\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:37.259805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.259800 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:37.259805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.259812 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafebcaf-aae2-43da-aac0-990cd46c9ecb-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:37.913589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.913560 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" Nov 02 06:32:37.913589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.913570 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm" event={"ID":"fafebcaf-aae2-43da-aac0-990cd46c9ecb","Type":"ContainerDied","Data":"b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b"} Nov 02 06:32:37.913848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:37.913618 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b42d6c42107c95154d39f0b8463b3506a40d9e2faa715bdcfe01999f9d67379b" Nov 02 06:32:38.552915 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:32:38.552882 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:32:44.273600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.273567 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:32:44.274121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.274096 6273 cpu_manager.go:414] "containerMap: removing container" podUID="fafebcaf-aae2-43da-aac0-990cd46c9ecb" containerName="revoke-keys" Nov 02 06:32:44.274121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.274113 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafebcaf-aae2-43da-aac0-990cd46c9ecb" containerName="revoke-keys" Nov 02 06:32:44.274231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.274220 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafebcaf-aae2-43da-aac0-990cd46c9ecb" containerName="revoke-keys" Nov 02 06:32:44.277584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.277562 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.326824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.326790 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-581608-openshift-default-dockercfg-bm4lm\"" Nov 02 06:32:44.375274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.375235 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:32:44.438026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.437971 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438133 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438166 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438183 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438311 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438350 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438372 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438511 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438492 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98v2\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.438582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.438565 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539392 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539450 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98v2\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539483 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539514 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539600 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.539719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539624 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539733 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539823 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539878 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.539919 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.540116 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.540171 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.540289 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.540359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.540295 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.541721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.541700 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.542224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.542204 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.593359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.593332 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.593568 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.593545 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98v2\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:44.598216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:44.598195 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:45.029419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.029387 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:32:45.075361 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:45.075321 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453e9a62_204d_4b39_8dd8_ca59f9235996.slice/crio-b5dcdd36f89bbd46362babba7e269392ad24c2b60a9d718e7f58f18c9db418e6 WatchSource:0}: Error finding container b5dcdd36f89bbd46362babba7e269392ad24c2b60a9d718e7f58f18c9db418e6: Status 404 returned error can't find the container with id b5dcdd36f89bbd46362babba7e269392ad24c2b60a9d718e7f58f18c9db418e6 Nov 02 06:32:45.078826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.078780 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x"] Nov 02 06:32:45.079083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.079049 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:45.079140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.079127 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:45.079179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.079167 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:45.084119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.084100 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.138875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.138833 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:45.139185 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.139152 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:45.141024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.140074 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x"] Nov 02 06:32:45.178325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.178290 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:32:45.182427 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.182405 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.222062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.222029 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:32:45.238303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.238273 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dcf9c3-openshift-default-dockercfg-7kbz6\"" Nov 02 06:32:45.250587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.250553 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.250894 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.250872 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swm4\" (UniqueName: \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.251140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.251115 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.352554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352475 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swm4\" (UniqueName: \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.352554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352520 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.352554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352543 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352565 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352719 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352748 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352791 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352807 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352905 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.352947 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.353086 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.353256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.353125 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fw62\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.353721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.353687 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.355152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.355135 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.406588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.406560 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swm4\" (UniqueName: \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\") pod \"maas-api-revoke-sub-e156d040-7fc4x\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.454284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454253 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.454284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454283 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.454477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454306 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.454477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454404 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.454477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454440 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.454477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454467 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454509 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454553 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454639 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fw62\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454713 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454737 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.454795 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.455078 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.455447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.455348 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.457204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.457174 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.457318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.457300 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.507938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.507901 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.508095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.508034 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fw62\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.700316 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.700283 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:45.796552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.796355 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:45.980742 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.980614 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" event={"ID":"453e9a62-204d-4b39-8dd8-ca59f9235996","Type":"ContainerStarted","Data":"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7"} Nov 02 06:32:45.980742 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:45.980691 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" event={"ID":"453e9a62-204d-4b39-8dd8-ca59f9235996","Type":"ContainerStarted","Data":"b5dcdd36f89bbd46362babba7e269392ad24c2b60a9d718e7f58f18c9db418e6"} Nov 02 06:32:46.094212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.094161 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podStartSLOduration=2.094145846 podStartE2EDuration="2.094145846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:46.093287026 +0000 UTC m=+920.066621710" watchObservedRunningTime="2026-11-02 06:32:46.094145846 +0000 UTC m=+920.067480530" Nov 02 06:32:46.126478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.126435 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x"] Nov 02 06:32:46.147624 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:46.147595 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488bf6f1_bea4_4da2_8482_3bbf4cd1930c.slice/crio-c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488 WatchSource:0}: Error finding container c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488: Status 404 returned error can't find the container with id c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488 Nov 02 06:32:46.205609 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.205582 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:32:46.215788 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.215763 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:32:46.222183 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:32:46.222157 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ace3267_1856_40b6_90ff_c6f4c541bab4.slice/crio-9e362beb3cd652f7f4222adfdf638184bd0e5d4fb0b44cc2e1a736fabb3e22ce WatchSource:0}: Error finding container 9e362beb3cd652f7f4222adfdf638184bd0e5d4fb0b44cc2e1a736fabb3e22ce: Status 404 returned error can't find the container with id 9e362beb3cd652f7f4222adfdf638184bd0e5d4fb0b44cc2e1a736fabb3e22ce Nov 02 06:32:46.226112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.225151 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:46.226112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.225239 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:46.226112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.225284 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:46.599845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.599818 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:46.601149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.601125 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:46.601296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.601171 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:46.990253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.990215 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" event={"ID":"0ace3267-1856-40b6-90ff-c6f4c541bab4","Type":"ContainerStarted","Data":"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0"} Nov 02 06:32:46.990427 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.990283 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" event={"ID":"0ace3267-1856-40b6-90ff-c6f4c541bab4","Type":"ContainerStarted","Data":"9e362beb3cd652f7f4222adfdf638184bd0e5d4fb0b44cc2e1a736fabb3e22ce"} Nov 02 06:32:46.993793 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.993761 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" event={"ID":"488bf6f1-bea4-4da2-8482-3bbf4cd1930c","Type":"ContainerStarted","Data":"2d84f8e741c820f762f3b16af5601769e442232612e8315ee2e365b4c3bb4ae7"} Nov 02 06:32:46.994027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:46.994008 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" event={"ID":"488bf6f1-bea4-4da2-8482-3bbf4cd1930c","Type":"ContainerStarted","Data":"c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488"} Nov 02 06:32:47.109850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.109795 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podStartSLOduration=2.109779286 podStartE2EDuration="2.109779286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:47.10844096 +0000 UTC m=+921.081775647" watchObservedRunningTime="2026-11-02 06:32:47.109779286 +0000 UTC m=+921.083113973" Nov 02 06:32:47.231044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.228039 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" podStartSLOduration=2.228012442 podStartE2EDuration="2.228012442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:47.22650748 +0000 UTC m=+921.199842167" watchObservedRunningTime="2026-11-02 06:32:47.228012442 +0000 UTC m=+921.201347137" Nov 02 06:32:47.601430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.601339 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:47.602429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.602395 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:47.803023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.799722 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:47.803023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.799779 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:47.803023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:47.800073 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:48.004675 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.004634 6273 generic.go:356] "Generic (PLEG): container finished" podID="488bf6f1-bea4-4da2-8482-3bbf4cd1930c" containerID="2d84f8e741c820f762f3b16af5601769e442232612e8315ee2e365b4c3bb4ae7" exitCode=0 Nov 02 06:32:48.004863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.004722 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" event={"ID":"488bf6f1-bea4-4da2-8482-3bbf4cd1930c","Type":"ContainerDied","Data":"2d84f8e741c820f762f3b16af5601769e442232612e8315ee2e365b4c3bb4ae7"} Nov 02 06:32:48.602887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.602416 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:48.602887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.602477 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:48.802010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.799358 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:48.802010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:48.799418 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:49.309574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.309482 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:49.347922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.347895 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\" (UniqueName: \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\") pod \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " Nov 02 06:32:49.348123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.348028 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\") pod \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " Nov 02 06:32:49.348199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.348175 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\") pod \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\" (UID: \"488bf6f1-bea4-4da2-8482-3bbf4cd1930c\") " Nov 02 06:32:49.379503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.379460 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca" pod "488bf6f1-bea4-4da2-8482-3bbf4cd1930c" (UID: "488bf6f1-bea4-4da2-8482-3bbf4cd1930c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:49.381238 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.381195 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca" pod "488bf6f1-bea4-4da2-8482-3bbf4cd1930c" (UID: "488bf6f1-bea4-4da2-8482-3bbf4cd1930c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:49.401488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.401431 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4" pod "488bf6f1-bea4-4da2-8482-3bbf4cd1930c" (UID: "488bf6f1-bea4-4da2-8482-3bbf4cd1930c"). InnerVolumeSpecName "kube-api-access-4swm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:49.474023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.473809 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:49.474023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.473847 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swm4\" (UniqueName: \"kubernetes.io/projected/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-kube-api-access-4swm4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:49.474023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.473865 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488bf6f1-bea4-4da2-8482-3bbf4cd1930c-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:49.610387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.610302 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:49.610387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.610357 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:49.801019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.800692 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:49.801019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:49.800753 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:50.018960 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.018927 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" Nov 02 06:32:50.019165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.018969 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x" event={"ID":"488bf6f1-bea4-4da2-8482-3bbf4cd1930c","Type":"ContainerDied","Data":"c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488"} Nov 02 06:32:50.019165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.019035 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5fb38dd4b940941a1a7ecb103e95277832008f76e33724641fef4de50366488" Nov 02 06:32:50.599477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.599429 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:50.599643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.599500 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:50.799122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.799081 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:50.799612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:50.799143 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:51.599368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:51.599332 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 06:32:51.599555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:51.599398 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 06:32:51.797596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:51.797566 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:51.797771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:51.797615 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:52.797136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:52.797098 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:32:52.797627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:52.797175 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 06:32:53.113857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.113770 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:53.114280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.114256 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:53.129531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.129505 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:32:53.801379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.801352 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:53.801791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.801771 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:32:53.802551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:32:53.802529 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:33:08.553036 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:33:08.552974 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:33:15.938407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:15.938361 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:33:15.938928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:15.938912 6273 cpu_manager.go:414] "containerMap: removing container" podUID="488bf6f1-bea4-4da2-8482-3bbf4cd1930c" containerName="revoke-keys" Nov 02 06:33:15.938928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:15.938927 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488bf6f1-bea4-4da2-8482-3bbf4cd1930c" containerName="revoke-keys" Nov 02 06:33:15.939093 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:15.939070 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="488bf6f1-bea4-4da2-8482-3bbf4cd1930c" containerName="revoke-keys" Nov 02 06:33:15.942877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:15.942856 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.019520 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.019492 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:33:16.039415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.039394 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"distinct-w3-fb1398-kserve-self-signed-certs\"" Nov 02 06:33:16.074174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074143 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.074294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074203 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqqz\" (UniqueName: \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.074339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074298 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.074380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074354 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.074520 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074502 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.074555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.074543 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175361 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175414 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175452 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175496 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqqz\" (UniqueName: \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175529 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175592 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.175946 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.175911 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.176152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.176130 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.176333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.176240 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.177910 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.177888 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.178159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.178139 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.229169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.229106 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqqz\" (UniqueName: \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\") pod \"distinct-w3-fb1398-kserve-587c84bd9f-5f622\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.255455 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.255428 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:16.656788 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:16.656752 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:33:16.664032 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:33:16.663976 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc57d7a7c_06ed_4975_aa69_bd2c15037394.slice/crio-5c3f7f32d878386a0d4f19059bf1dabb4cfefcba63d40be4660cdea3fb350d3b WatchSource:0}: Error finding container 5c3f7f32d878386a0d4f19059bf1dabb4cfefcba63d40be4660cdea3fb350d3b: Status 404 returned error can't find the container with id 5c3f7f32d878386a0d4f19059bf1dabb4cfefcba63d40be4660cdea3fb350d3b Nov 02 06:33:17.199500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:17.199466 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" event={"ID":"c57d7a7c-06ed-4975-aa69-bd2c15037394","Type":"ContainerStarted","Data":"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07"} Nov 02 06:33:17.199930 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:17.199507 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" event={"ID":"c57d7a7c-06ed-4975-aa69-bd2c15037394","Type":"ContainerStarted","Data":"5c3f7f32d878386a0d4f19059bf1dabb4cfefcba63d40be4660cdea3fb350d3b"} Nov 02 06:33:17.256148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:17.256119 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:17.308333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:17.308289 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podStartSLOduration=2.308274249 podStartE2EDuration="2.308274249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:17.306495411 +0000 UTC m=+951.279830110" watchObservedRunningTime="2026-11-02 06:33:17.308274249 +0000 UTC m=+951.281608933" Nov 02 06:33:19.000099 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:19.000065 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:33:19.000529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:19.000351 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" containerID="cri-o://02c834c6b4e2b69e9562cc156c78d2ecaaca0bb8bc162ad585444abf0343c7cf" gracePeriod=30 Nov 02 06:33:19.039343 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:19.039309 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-htmtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:33:19.039460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:19.039373 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:33:24.239957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.239928 6273 generic.go:356] "Generic (PLEG): container finished" podID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerID="02c834c6b4e2b69e9562cc156c78d2ecaaca0bb8bc162ad585444abf0343c7cf" exitCode=0 Nov 02 06:33:24.240351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.239973 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" event={"ID":"855763ec-df91-40f7-a0c2-3059e2ea2c24","Type":"ContainerDied","Data":"02c834c6b4e2b69e9562cc156c78d2ecaaca0bb8bc162ad585444abf0343c7cf"} Nov 02 06:33:24.265512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.265458 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:33:24.360601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360567 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360610 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360650 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360710 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360747 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360859 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.360923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360914 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.361114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.360968 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.361114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361042 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\") pod \"855763ec-df91-40f7-a0c2-3059e2ea2c24\" (UID: \"855763ec-df91-40f7-a0c2-3059e2ea2c24\") " Nov 02 06:33:24.361217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361111 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:33:24.361217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361121 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:33:24.361217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361126 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:33:24.361448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361425 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:33:24.361510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361439 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:33:24.361651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361614 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.361651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361642 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855763ec-df91-40f7-a0c2-3059e2ea2c24-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.361844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361657 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.361844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361672 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.361844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.361686 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.363024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.362970 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:33:24.363160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.363140 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "kube-api-access-fd2ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:33:24.363237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.363218 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:33:24.363306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.363276 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo" pod "855763ec-df91-40f7-a0c2-3059e2ea2c24" (UID: "855763ec-df91-40f7-a0c2-3059e2ea2c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:33:24.463483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.463446 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.463626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.463486 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.463626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.463501 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-kube-api-access-fd2ml\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:24.463626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:24.463516 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855763ec-df91-40f7-a0c2-3059e2ea2c24-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:25.247082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:25.247051 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" Nov 02 06:33:25.247548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:25.247080 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx" event={"ID":"855763ec-df91-40f7-a0c2-3059e2ea2c24","Type":"ContainerDied","Data":"a5c6d8eb116ab17131b6c5c547c0557cd724acf760b3d4d4d8e3166f3f661a19"} Nov 02 06:33:25.247548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:25.247151 6273 scope.go:122] "RemoveContainer" containerID="02c834c6b4e2b69e9562cc156c78d2ecaaca0bb8bc162ad585444abf0343c7cf" Nov 02 06:33:25.447914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:25.447877 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:33:25.494464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:25.494427 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-htmtx"] Nov 02 06:33:26.179279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:26.179246 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" path="/var/lib/kubelet/pods/855763ec-df91-40f7-a0c2-3059e2ea2c24/volumes" Nov 02 06:33:26.255973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:26.255944 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:26.268595 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:26.268568 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:27.268211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:27.268183 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:33:29.483168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:29.483134 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:33:32.233706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.233670 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 06:33:32.234312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.234292 6273 cpu_manager.go:414] "containerMap: removing container" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" Nov 02 06:33:32.234312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.234313 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" Nov 02 06:33:32.234453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.234443 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="855763ec-df91-40f7-a0c2-3059e2ea2c24" containerName="istio-proxy" Nov 02 06:33:32.239253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.239231 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.251711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.251687 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.251828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.251765 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.252171 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.252146 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.252255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.252186 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.252319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.252295 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfh2k\" (UniqueName: \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.252375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.252355 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.296797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.296763 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 06:33:32.334885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.334858 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"kube-root-ca.crt\"" Nov 02 06:33:32.335050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.334927 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"openshift-service-ca.crt\"" Nov 02 06:33:32.335050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.334981 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"default-dockercfg-28q6p\"" Nov 02 06:33:32.335182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.335163 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"distinct-w1-cdfce9-kserve-self-signed-certs\"" Nov 02 06:33:32.353604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353575 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.353752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353612 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.353752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353664 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfh2k\" (UniqueName: \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.353752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353707 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.353928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353807 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.353928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.353861 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.354134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.354112 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.354207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.354179 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.354323 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.354299 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.356075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.356046 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.356362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.356343 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.408791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.408761 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfh2k\" (UniqueName: \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\") pod \"distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.553098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.553017 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:32.938664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:32.938630 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 06:33:32.945148 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:33:32.945122 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1146e3e_9db0_40fb_8b6a_99d8398524e4.slice/crio-c880b5bcf699eb4ee344b78493bad98c3d3d5bad99f90f1f8354c37dab5e5006 WatchSource:0}: Error finding container c880b5bcf699eb4ee344b78493bad98c3d3d5bad99f90f1f8354c37dab5e5006: Status 404 returned error can't find the container with id c880b5bcf699eb4ee344b78493bad98c3d3d5bad99f90f1f8354c37dab5e5006 Nov 02 06:33:33.304224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:33.304129 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" event={"ID":"c1146e3e-9db0-40fb-8b6a-99d8398524e4","Type":"ContainerStarted","Data":"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6"} Nov 02 06:33:33.304224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:33.304172 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" event={"ID":"c1146e3e-9db0-40fb-8b6a-99d8398524e4","Type":"ContainerStarted","Data":"c880b5bcf699eb4ee344b78493bad98c3d3d5bad99f90f1f8354c37dab5e5006"} Nov 02 06:33:33.411723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:33.411678 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podStartSLOduration=1.411665317 podStartE2EDuration="1.411665317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:33.410417426 +0000 UTC m=+967.383752110" watchObservedRunningTime="2026-11-02 06:33:33.411665317 +0000 UTC m=+967.385000000" Nov 02 06:33:33.553830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:33.553797 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:38.551795 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:33:38.551748 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:33:42.553461 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:42.553421 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:42.566213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:42.566181 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:43.372057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:43.372020 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:33:59.324578 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.324542 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:33:59.329168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.329148 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356105 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356070 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487kn\" (UniqueName: \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356436 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356408 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356447 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356567 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356605 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.356658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.356628 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.426235 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.426200 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"distinct-w5-dcf9c3-kserve-self-signed-certs\"" Nov 02 06:33:59.428575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.428551 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:33:59.457782 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.457742 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.457933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.457803 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.457933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.457824 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.457933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.457892 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487kn\" (UniqueName: \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.458152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.457999 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.458152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.458026 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.458258 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.458170 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.458258 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.458214 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.458380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.458363 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.460244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.460220 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.461308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.461288 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.512609 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.512578 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487kn\" (UniqueName: \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\") pod \"distinct-w5-dcf9c3-kserve-c567b87db-574s9\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:33:59.642024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:33:59.641925 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:34:00.037763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:00.037727 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:34:00.045135 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:34:00.045108 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0abeaf2_d06a_44bf_87b9_50261b9c95a8.slice/crio-211b8c28ec4284d0c770c7a4bdde858edea9e6b57de2abfb1b3b8361cbba70ba WatchSource:0}: Error finding container 211b8c28ec4284d0c770c7a4bdde858edea9e6b57de2abfb1b3b8361cbba70ba: Status 404 returned error can't find the container with id 211b8c28ec4284d0c770c7a4bdde858edea9e6b57de2abfb1b3b8361cbba70ba Nov 02 06:34:00.458186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:00.458150 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" event={"ID":"b0abeaf2-d06a-44bf-87b9-50261b9c95a8","Type":"ContainerStarted","Data":"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a"} Nov 02 06:34:00.458186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:00.458194 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" event={"ID":"b0abeaf2-d06a-44bf-87b9-50261b9c95a8","Type":"ContainerStarted","Data":"211b8c28ec4284d0c770c7a4bdde858edea9e6b57de2abfb1b3b8361cbba70ba"} Nov 02 06:34:00.567979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:00.567919 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podStartSLOduration=1.567903162 podStartE2EDuration="1.567903162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:34:00.566149078 +0000 UTC m=+994.539483786" watchObservedRunningTime="2026-11-02 06:34:00.567903162 +0000 UTC m=+994.541237846" Nov 02 06:34:00.642848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:00.642814 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:34:08.394269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:08.394172 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:34:08.552500 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:34:08.552455 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:34:09.642698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:09.642651 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:34:09.659028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:09.658979 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:34:10.526589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:10.526558 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:34:26.056781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.056736 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 06:34:26.061065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.061043 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.157922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.157889 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"distinct-2-w6-6ff85e-kserve-self-signed-certs\"" Nov 02 06:34:26.158256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158220 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.158376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158266 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.158443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158395 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.158506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158486 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.158579 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158561 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt59\" (UniqueName: \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.158637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.158615 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.160260 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.160240 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 06:34:26.259424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259393 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259448 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt59\" (UniqueName: \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259505 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259623 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259669 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259727 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.259974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.259950 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.260100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.260071 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.260241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.260129 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.261830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.261803 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.262196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.262175 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.312606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.312547 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt59\" (UniqueName: \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\") pod \"distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.373534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.373509 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:26.746511 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:26.746477 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 06:34:26.753042 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:34:26.753014 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b637f7d_0cb9_49ce_8412_8ca46c3be9fb.slice/crio-90595d3915548564c3a63bc10a02bca1248b6910b1440c3bcf9ece6ad21d9edc WatchSource:0}: Error finding container 90595d3915548564c3a63bc10a02bca1248b6910b1440c3bcf9ece6ad21d9edc: Status 404 returned error can't find the container with id 90595d3915548564c3a63bc10a02bca1248b6910b1440c3bcf9ece6ad21d9edc Nov 02 06:34:27.627931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:27.627888 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" event={"ID":"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb","Type":"ContainerStarted","Data":"cdaf8dc3805c75a7cbafa6f01633d79b3754ca5b3e4e5a07ca1ba57b457aadd2"} Nov 02 06:34:27.628332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:27.627940 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" event={"ID":"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb","Type":"ContainerStarted","Data":"90595d3915548564c3a63bc10a02bca1248b6910b1440c3bcf9ece6ad21d9edc"} Nov 02 06:34:27.737464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:27.737413 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podStartSLOduration=1.737399667 podStartE2EDuration="1.737399667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:34:27.735922325 +0000 UTC m=+1021.709257011" watchObservedRunningTime="2026-11-02 06:34:27.737399667 +0000 UTC m=+1021.710734352" Nov 02 06:34:28.374174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:28.374140 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:36.373836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:36.373802 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:36.386135 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:36.386109 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:36.695257 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:36.695219 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:34:38.554044 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:34:38.554008 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:34:43.634702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:34:43.634669 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:35:01.702122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.702088 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:35:01.706468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.706445 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748438 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748569 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748630 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748676 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748754 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ffw\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748791 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748887 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.748961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.748914 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.749065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.749043 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.755352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.755334 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6ff85e-openshift-default-dockercfg-f2wx5\"" Nov 02 06:35:01.805623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.805590 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:35:01.850460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850429 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850463 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850495 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850528 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850577 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850614 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850682 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850750 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ffw\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850809 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.850928 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.850979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.851003 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.851340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.851293 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.851340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.851316 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.851476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.851351 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.853060 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.853035 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.853257 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.853241 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.904124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.904097 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:01.904777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:01.904753 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ffw\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:02.020478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.020392 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:02.451240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.451206 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:35:02.458049 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:35:02.458017 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9033df_a130_4d5a_8037_b416d78c4e19.slice/crio-abd39bc6bb926dceaa988c7d080906de9c85f5058eb3d75dcb1a47601e26c9ce WatchSource:0}: Error finding container abd39bc6bb926dceaa988c7d080906de9c85f5058eb3d75dcb1a47601e26c9ce: Status 404 returned error can't find the container with id abd39bc6bb926dceaa988c7d080906de9c85f5058eb3d75dcb1a47601e26c9ce Nov 02 06:35:02.460419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.460384 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.460518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.460453 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.460518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.460483 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.841559 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.841471 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" event={"ID":"7b9033df-a130-4d5a-8037-b416d78c4e19","Type":"ContainerStarted","Data":"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0"} Nov 02 06:35:02.841559 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.841531 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" event={"ID":"7b9033df-a130-4d5a-8037-b416d78c4e19","Type":"ContainerStarted","Data":"abd39bc6bb926dceaa988c7d080906de9c85f5058eb3d75dcb1a47601e26c9ce"} Nov 02 06:35:02.953040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:02.952958 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podStartSLOduration=1.952941621 podStartE2EDuration="1.952941621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:02.950531492 +0000 UTC m=+1056.923866175" watchObservedRunningTime="2026-11-02 06:35:02.952941621 +0000 UTC m=+1056.926276305" Nov 02 06:35:03.020707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:03.020673 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:03.022193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:03.022167 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:03.022311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:03.022222 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:04.021223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:04.021189 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:04.021599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:04.021246 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:05.021161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:05.021121 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:05.021340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:05.021185 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:06.020978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:06.020945 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:06.021164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:06.021030 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:07.021667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:07.021631 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:07.022064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:07.021714 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:08.020858 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:08.020823 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:08.021055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:08.020879 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:08.553879 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:35:08.553836 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:35:09.021458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:09.021425 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 06:35:09.021634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:09.021484 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 06:35:10.024977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:10.024943 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:10.025518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:10.025381 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:10.026113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:10.026090 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:35:12.478978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:12.478945 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:35:38.554250 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:35:38.554174 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:35:39.158713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.158679 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:35:39.163114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.163093 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.259906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.259880 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"openshift-service-ca.crt\"" Nov 02 06:35:39.259906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.259895 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"kube-root-ca.crt\"" Nov 02 06:35:39.260172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.259945 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"distinct-2-w4-680b7b-kserve-self-signed-certs\"" Nov 02 06:35:39.260369 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.260354 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"default-dockercfg-8rnrh\"" Nov 02 06:35:39.262342 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.262322 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:35:39.279909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.279888 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.280054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.279919 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.280114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.280095 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.280195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.280183 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.280232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.280209 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfz5\" (UniqueName: \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.280265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.280231 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.380961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.380929 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.380972 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfz5\" (UniqueName: \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381040 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381107 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381141 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381261 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381506 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381587 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.381674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.381603 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.383375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.383348 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.383593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.383573 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.434607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.434543 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfz5\" (UniqueName: \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\") pod \"distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.475828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.475803 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:39.867771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:39.867636 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:35:39.874038 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:35:39.874007 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd22a27e0_6ea9_4d49_9d3b_215a10dc074c.slice/crio-6d9587377a0c8389956f0b182059b61fae521389776d859dca77a39c4479e515 WatchSource:0}: Error finding container 6d9587377a0c8389956f0b182059b61fae521389776d859dca77a39c4479e515: Status 404 returned error can't find the container with id 6d9587377a0c8389956f0b182059b61fae521389776d859dca77a39c4479e515 Nov 02 06:35:40.086033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:40.085976 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" event={"ID":"d22a27e0-6ea9-4d49-9d3b-215a10dc074c","Type":"ContainerStarted","Data":"68b78ec0cfba665594bacdf8a86d06e782d856fb11fcb3798adbb36ef91c9439"} Nov 02 06:35:40.086033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:40.086039 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" event={"ID":"d22a27e0-6ea9-4d49-9d3b-215a10dc074c","Type":"ContainerStarted","Data":"6d9587377a0c8389956f0b182059b61fae521389776d859dca77a39c4479e515"} Nov 02 06:35:40.193590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:40.193526 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podStartSLOduration=1.193513718 podStartE2EDuration="1.193513718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:40.192770832 +0000 UTC m=+1094.166105517" watchObservedRunningTime="2026-11-02 06:35:40.193513718 +0000 UTC m=+1094.166848401" Nov 02 06:35:40.476816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:40.476738 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:42.535594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:42.535562 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:35:49.476840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:49.476804 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:49.489277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:49.489253 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:35:50.158104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:35:50.158070 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:36:01.265377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.265341 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:36:01.270081 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.270056 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.323740 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.323702 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:36:01.431784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.431742 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.431980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.431859 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.431980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.431895 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.431980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.431916 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.432181 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.432058 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.432181 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.432099 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.432290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.432200 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcs8\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.432290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.432224 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.432290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.432247 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533468 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533529 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533566 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533598 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533644 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533682 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533763 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcs8\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533792 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533831 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.533935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533899 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.534167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.533973 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.534258 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.534194 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.534321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.534268 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.534357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.534331 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.535910 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.535888 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.536056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.536037 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.587389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.587347 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.587553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.587539 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcs8\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:01.885328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:01.885232 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:02.319416 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:02.319379 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:36:02.326066 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:02.326033 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75537f40_6928_4c3a_b0ab_5879184a9d7e.slice/crio-eb04f317dc58d62aae3142defc7a7afe0f27dd12ee109c5839fb71e7f4cb637b WatchSource:0}: Error finding container eb04f317dc58d62aae3142defc7a7afe0f27dd12ee109c5839fb71e7f4cb637b: Status 404 returned error can't find the container with id eb04f317dc58d62aae3142defc7a7afe0f27dd12ee109c5839fb71e7f4cb637b Nov 02 06:36:02.329362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:02.329315 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:02.329473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:02.329411 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:02.329473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:02.329452 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:03.224447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.224402 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" event={"ID":"75537f40-6928-4c3a-b0ab-5879184a9d7e","Type":"ContainerStarted","Data":"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f"} Nov 02 06:36:03.224621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.224479 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" event={"ID":"75537f40-6928-4c3a-b0ab-5879184a9d7e","Type":"ContainerStarted","Data":"eb04f317dc58d62aae3142defc7a7afe0f27dd12ee109c5839fb71e7f4cb637b"} Nov 02 06:36:03.439049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.438968 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podStartSLOduration=2.438946674 podStartE2EDuration="2.438946674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:03.392080652 +0000 UTC m=+1117.365415340" watchObservedRunningTime="2026-11-02 06:36:03.438946674 +0000 UTC m=+1117.412281359" Nov 02 06:36:03.885562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.885529 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:03.887307 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.887280 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:03.887443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:03.887325 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:04.885843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:04.885811 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:04.886242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:04.885868 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:05.885598 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:05.885562 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:05.885809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:05.885621 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:06.886253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:06.886216 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:06.886656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:06.886275 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:07.886269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:07.886233 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:07.886656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:07.886290 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:08.554050 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:36:08.554004 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:36:08.885867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:08.885782 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 06:36:08.885867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:08.885845 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 06:36:09.890620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:09.890582 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:09.891233 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:09.890968 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:09.891723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:09.891704 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:36:15.660261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.660218 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:36:15.664704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.664681 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735181 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735234 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735348 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735367 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735393 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735408 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q927l\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735530 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735551 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.735596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.735567 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.763182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.763154 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:36:15.836349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836316 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836353 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836381 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836397 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q927l\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836431 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836455 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836471 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836615 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.836882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836662 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.837029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.836952 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.837095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.837040 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.837159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.837116 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.837159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.837123 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.837319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.837299 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.838801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.838777 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.838940 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.838923 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.891952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.891921 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.892244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.892221 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q927l\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:15.978604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:15.978541 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:16.403828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:16.403797 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:36:16.409873 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:16.409848 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861873a0_3f3a_4bbf_81eb_28c412173941.slice/crio-673e127859d9e3b2230c396120a00e0c8ac4f32fbe8a4b385640634142c04054 WatchSource:0}: Error finding container 673e127859d9e3b2230c396120a00e0c8ac4f32fbe8a4b385640634142c04054: Status 404 returned error can't find the container with id 673e127859d9e3b2230c396120a00e0c8ac4f32fbe8a4b385640634142c04054 Nov 02 06:36:16.412229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:16.412197 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:16.412295 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:16.412261 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:16.412295 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:16.412288 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:17.333174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.333138 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" event={"ID":"861873a0-3f3a-4bbf-81eb-28c412173941","Type":"ContainerStarted","Data":"b7605c27792e3515a40ecfe3c42bf619272647872cfb25ccafa3b02492b91225"} Nov 02 06:36:17.333608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.333188 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" event={"ID":"861873a0-3f3a-4bbf-81eb-28c412173941","Type":"ContainerStarted","Data":"673e127859d9e3b2230c396120a00e0c8ac4f32fbe8a4b385640634142c04054"} Nov 02 06:36:17.515534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.515474 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podStartSLOduration=2.515446034 podStartE2EDuration="2.515446034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:17.490610035 +0000 UTC m=+1131.463944772" watchObservedRunningTime="2026-11-02 06:36:17.515446034 +0000 UTC m=+1131.488780720" Nov 02 06:36:17.979359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.979323 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:17.981022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.980981 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:17.981130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:17.981045 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:18.979349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:18.979311 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:18.979754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:18.979381 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:19.978927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:19.978894 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:19.979169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:19.979132 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:20.979538 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:20.979505 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:20.980151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:20.979579 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:21.979826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:21.979790 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:21.980266 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:21.979849 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:22.978838 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:22.978801 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:36:22.979039 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:22.978852 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:36:23.983121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:23.983083 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:23.983598 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:23.983373 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:23.984128 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:23.984112 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:36:31.787517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:31.787478 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:36:36.044361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:36.044327 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84"] Nov 02 06:36:36.091467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:36.091433 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf28a2f-znp84"] Nov 02 06:36:36.180040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:36.179979 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9fbf207-0e2d-4ec9-9b58-6df8998dce2c" path="/var/lib/kubelet/pods/f9fbf207-0e2d-4ec9-9b58-6df8998dce2c/volumes" Nov 02 06:36:37.322769 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.322736 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g"] Nov 02 06:36:37.328130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.328112 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.376562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.376532 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvlc\" (UniqueName: \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.376692 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.376588 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.376781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.376757 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.379654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.379631 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g"] Nov 02 06:36:37.422968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.422943 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:37.423104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.423013 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:37.477729 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.477702 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.477867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.477815 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvlc\" (UniqueName: \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.477867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.477839 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.478443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.478421 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.480046 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.480025 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.531465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.531439 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvlc\" (UniqueName: \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\") pod \"maas-api-revoke-sub-a5e9465b-2pm8g\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:37.641746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:37.641676 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:38.013003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:38.012938 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g"] Nov 02 06:36:38.019002 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:38.018960 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a762ea5_ef86_4159_9e65_3c8ec57e8143.slice/crio-b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766 WatchSource:0}: Error finding container b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766: Status 404 returned error can't find the container with id b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766 Nov 02 06:36:38.553827 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:36:38.553792 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:36:38.563502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:38.563468 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" event={"ID":"4a762ea5-ef86-4159-9e65-3c8ec57e8143","Type":"ContainerStarted","Data":"c9ed639faacfb285d96191c2a75caaa8725f1c101b6fb13e3e088a99b87fdcd9"} Nov 02 06:36:38.563663 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:38.563532 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" event={"ID":"4a762ea5-ef86-4159-9e65-3c8ec57e8143","Type":"ContainerStarted","Data":"b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766"} Nov 02 06:36:38.672976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:38.672924 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" podStartSLOduration=1.6729112769999999 podStartE2EDuration="1.672911277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:38.670339856 +0000 UTC m=+1152.643674540" watchObservedRunningTime="2026-11-02 06:36:38.672911277 +0000 UTC m=+1152.646245951" Nov 02 06:36:39.570724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:39.570684 6273 generic.go:356] "Generic (PLEG): container finished" podID="4a762ea5-ef86-4159-9e65-3c8ec57e8143" containerID="c9ed639faacfb285d96191c2a75caaa8725f1c101b6fb13e3e088a99b87fdcd9" exitCode=0 Nov 02 06:36:39.571212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:39.570770 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" event={"ID":"4a762ea5-ef86-4159-9e65-3c8ec57e8143","Type":"ContainerDied","Data":"c9ed639faacfb285d96191c2a75caaa8725f1c101b6fb13e3e088a99b87fdcd9"} Nov 02 06:36:40.733841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.733812 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:40.818607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.818579 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\") pod \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " Nov 02 06:36:40.818776 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.818723 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\" (UniqueName: \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\") pod \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " Nov 02 06:36:40.818776 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.818743 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\") pod \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\" (UID: \"4a762ea5-ef86-4159-9e65-3c8ec57e8143\") " Nov 02 06:36:40.818981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.818956 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca" pod "4a762ea5-ef86-4159-9e65-3c8ec57e8143" (UID: "4a762ea5-ef86-4159-9e65-3c8ec57e8143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:40.819133 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.819116 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.820836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.820804 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca" pod "4a762ea5-ef86-4159-9e65-3c8ec57e8143" (UID: "4a762ea5-ef86-4159-9e65-3c8ec57e8143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:40.820933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.820898 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc" pod "4a762ea5-ef86-4159-9e65-3c8ec57e8143" (UID: "4a762ea5-ef86-4159-9e65-3c8ec57e8143"). InnerVolumeSpecName "kube-api-access-7vvlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:40.919913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.919882 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvlc\" (UniqueName: \"kubernetes.io/projected/4a762ea5-ef86-4159-9e65-3c8ec57e8143-kube-api-access-7vvlc\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.919913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:40.919911 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a762ea5-ef86-4159-9e65-3c8ec57e8143-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.434522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.434491 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld"] Nov 02 06:36:41.435104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.435087 6273 cpu_manager.go:414] "containerMap: removing container" podUID="4a762ea5-ef86-4159-9e65-3c8ec57e8143" containerName="revoke-keys" Nov 02 06:36:41.435104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.435104 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a762ea5-ef86-4159-9e65-3c8ec57e8143" containerName="revoke-keys" Nov 02 06:36:41.435265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.435236 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a762ea5-ef86-4159-9e65-3c8ec57e8143" containerName="revoke-keys" Nov 02 06:36:41.438598 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.438579 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.491392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.491362 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld"] Nov 02 06:36:41.526258 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.526223 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.526439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.526286 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.526512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.526445 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4zmd\" (UniqueName: \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.583920 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.583889 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" Nov 02 06:36:41.584104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.583893 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g" event={"ID":"4a762ea5-ef86-4159-9e65-3c8ec57e8143","Type":"ContainerDied","Data":"b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766"} Nov 02 06:36:41.584104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.584034 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3f0bb84b7576d1c928e6e6570c3a5d1a70b72e5b00d5b0b98c768bd5cbe8766" Nov 02 06:36:41.627298 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.627270 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.627426 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.627352 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4zmd\" (UniqueName: \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.627537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.627519 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.627970 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.627947 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.629868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.629844 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.681110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.681085 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4zmd\" (UniqueName: \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\") pod \"maas-api-revoke-sub-9bbdb6ee-gq7ld\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:41.751915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:41.751854 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:42.122946 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:42.122851 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld"] Nov 02 06:36:42.129663 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:42.129628 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adf0f40_d9a6_4e2f_8c77_8c50263b1a62.slice/crio-5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4 WatchSource:0}: Error finding container 5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4: Status 404 returned error can't find the container with id 5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4 Nov 02 06:36:42.592065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:42.592025 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" event={"ID":"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62","Type":"ContainerStarted","Data":"a0b4f6f60ed65e0f2e1786c2098640578fcf53b11700ce64aad504c2b9b1d8d5"} Nov 02 06:36:42.592495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:42.592079 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" event={"ID":"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62","Type":"ContainerStarted","Data":"5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4"} Nov 02 06:36:42.701389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:42.701335 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" podStartSLOduration=1.7013214749999999 podStartE2EDuration="1.701321475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:42.698434449 +0000 UTC m=+1156.671769152" watchObservedRunningTime="2026-11-02 06:36:42.701321475 +0000 UTC m=+1156.674656158" Nov 02 06:36:43.599628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:43.599593 6273 generic.go:356] "Generic (PLEG): container finished" podID="6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" containerID="a0b4f6f60ed65e0f2e1786c2098640578fcf53b11700ce64aad504c2b9b1d8d5" exitCode=0 Nov 02 06:36:43.600093 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:43.599681 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" event={"ID":"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62","Type":"ContainerDied","Data":"a0b4f6f60ed65e0f2e1786c2098640578fcf53b11700ce64aad504c2b9b1d8d5"} Nov 02 06:36:44.761842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.761815 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:44.865348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.865270 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\") pod \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " Nov 02 06:36:44.865487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.865395 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\" (UniqueName: \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\") pod \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " Nov 02 06:36:44.865529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.865503 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\") pod \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\" (UID: \"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62\") " Nov 02 06:36:44.865831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.865804 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca" pod "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" (UID: "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:44.866062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.866046 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:44.868338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.867631 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd" pod "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" (UID: "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62"). InnerVolumeSpecName "kube-api-access-p4zmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:44.868338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.867756 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca" pod "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" (UID: "6adf0f40-d9a6-4e2f-8c77-8c50263b1a62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:44.966955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.966924 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:44.966955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:44.966953 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4zmd\" (UniqueName: \"kubernetes.io/projected/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62-kube-api-access-p4zmd\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:45.615825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:45.615787 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" event={"ID":"6adf0f40-d9a6-4e2f-8c77-8c50263b1a62","Type":"ContainerDied","Data":"5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4"} Nov 02 06:36:45.616027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:45.615853 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5220bb43362fbed1bdd4363b221e5dd4f3aca6b96adb5746dd6dc4502e489dc4" Nov 02 06:36:45.616027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:45.615802 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld" Nov 02 06:36:46.598419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.598388 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:36:46.599031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.599014 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" containerName="revoke-keys" Nov 02 06:36:46.599031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.599031 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" containerName="revoke-keys" Nov 02 06:36:46.599152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.599133 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" containerName="revoke-keys" Nov 02 06:36:46.602709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.602691 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.688809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.688780 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.689014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.688831 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.689014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.688902 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.689168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.689016 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.689168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.689092 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5874\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.689293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.689176 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.690012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.689947 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.690209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.690124 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.690332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.690313 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.703803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.703774 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:36:46.791874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.791845 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.791893 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5874\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.791928 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792047 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792096 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792161 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792258 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792381 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792421 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792436 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792513 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792637 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792661 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.792792 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.792771 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.794361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.794339 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.794455 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.794416 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.855758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.855693 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5874\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.856185 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.856165 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.911999 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.911967 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:36:46.916182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.916157 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:46.917350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.917336 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:46.972939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:46.972904 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:36:47.115903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.115805 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.115903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.115853 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.115936 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.115956 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.115982 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.116088 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.116139 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116194 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.116155 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nx4\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.116477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.116206 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217750 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217812 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217840 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217886 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217914 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.217962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.217958 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218095 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218180 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218203 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nx4\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218316 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218354 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218398 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218437 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.218910 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.218887 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.220269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.220247 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.220328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.220310 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.280243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.280222 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.280334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.280249 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nx4\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.304981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.304958 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:36:47.309441 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:47.309420 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362133a2_bb5b_4d12_a639_db8ab15ee1a8.slice/crio-88018d269d4d071c15be752c5733f39326679c8e2f1e0a906aa8a2714dd13d9d WatchSource:0}: Error finding container 88018d269d4d071c15be752c5733f39326679c8e2f1e0a906aa8a2714dd13d9d: Status 404 returned error can't find the container with id 88018d269d4d071c15be752c5733f39326679c8e2f1e0a906aa8a2714dd13d9d Nov 02 06:36:47.311949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.311917 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.312070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.312008 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.312070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.312058 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.546874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.546841 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:47.639881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.638462 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" event={"ID":"362133a2-bb5b-4d12-a639-db8ab15ee1a8","Type":"ContainerStarted","Data":"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f"} Nov 02 06:36:47.639881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.638515 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" event={"ID":"362133a2-bb5b-4d12-a639-db8ab15ee1a8","Type":"ContainerStarted","Data":"88018d269d4d071c15be752c5733f39326679c8e2f1e0a906aa8a2714dd13d9d"} Nov 02 06:36:47.755380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.755324 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podStartSLOduration=1.755309618 podStartE2EDuration="1.755309618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:47.754003404 +0000 UTC m=+1161.727338087" watchObservedRunningTime="2026-11-02 06:36:47.755309618 +0000 UTC m=+1161.728644301" Nov 02 06:36:47.917339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.917309 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:47.935009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.934968 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:36:47.941967 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:47.941937 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f46032_62d2_4d9e_88be_16e246168c97.slice/crio-4fe8a5521f142ed9c423cb515eaff5514102f6834407e6452d17f4cc57b852fa WatchSource:0}: Error finding container 4fe8a5521f142ed9c423cb515eaff5514102f6834407e6452d17f4cc57b852fa: Status 404 returned error can't find the container with id 4fe8a5521f142ed9c423cb515eaff5514102f6834407e6452d17f4cc57b852fa Nov 02 06:36:47.944374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.944341 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.944462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.944407 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.944462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:47.944438 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:48.652015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:48.647921 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" event={"ID":"98f46032-62d2-4d9e-88be-16e246168c97","Type":"ContainerStarted","Data":"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f"} Nov 02 06:36:48.652015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:48.647970 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" event={"ID":"98f46032-62d2-4d9e-88be-16e246168c97","Type":"ContainerStarted","Data":"4fe8a5521f142ed9c423cb515eaff5514102f6834407e6452d17f4cc57b852fa"} Nov 02 06:36:48.802882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:48.802759 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podStartSLOduration=2.802740252 podStartE2EDuration="2.802740252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:48.800394413 +0000 UTC m=+1162.773729098" watchObservedRunningTime="2026-11-02 06:36:48.802740252 +0000 UTC m=+1162.776074938" Nov 02 06:36:48.917906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:48.917822 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:48.917906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:48.917877 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:49.547915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:49.547886 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:49.570013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:49.558121 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.570013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:49.558199 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:49.917475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:49.917442 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.917953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:49.917507 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:50.547392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:50.547346 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.547587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:50.547419 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:50.916686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:50.916636 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.916867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:50.916697 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:51.547526 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:51.547260 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.547526 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:51.547338 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:51.917000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:51.916961 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.917170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:51.917057 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:52.022965 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.022930 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr"] Nov 02 06:36:52.027481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.027458 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.104014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.103161 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:52.130423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.130387 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.130594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.130570 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gpf\" (UniqueName: \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.130653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.130630 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.183070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.182775 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:52.211830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.211798 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr"] Nov 02 06:36:52.232374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.232344 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.232523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.232462 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gpf\" (UniqueName: \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.232523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.232505 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.233136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.233112 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.234763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.234745 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.302757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.302718 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gpf\" (UniqueName: \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\") pod \"maas-api-revoke-sub-74d2eef6-pvjkr\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.341347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.341320 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:52.547226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.547198 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.547430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.547409 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:52.806050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.805857 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr"] Nov 02 06:36:52.811738 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:36:52.811711 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51213ce_c8a3_4956_ab31_f5ba9de58395.slice/crio-9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df WatchSource:0}: Error finding container 9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df: Status 404 returned error can't find the container with id 9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df Nov 02 06:36:52.917681 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.917650 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.917859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:52.917706 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:53.547566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.547533 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.547738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.547584 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:53.726403 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.726363 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" event={"ID":"e51213ce-c8a3-4956-ab31-f5ba9de58395","Type":"ContainerStarted","Data":"69f81c20a4c09c8a374132a2bbf50ba375cd6708c0cf913b2a598000760ce7a7"} Nov 02 06:36:53.726610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.726430 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" event={"ID":"e51213ce-c8a3-4956-ab31-f5ba9de58395","Type":"ContainerStarted","Data":"9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df"} Nov 02 06:36:53.838038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.837914 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" podStartSLOduration=2.837895991 podStartE2EDuration="2.837895991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:53.836109317 +0000 UTC m=+1167.809444016" watchObservedRunningTime="2026-11-02 06:36:53.837895991 +0000 UTC m=+1167.811230677" Nov 02 06:36:53.916911 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.916873 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.917097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:53.916928 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 06:36:54.547487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.547447 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 06:36:54.547645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.547515 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 06:36:54.737896 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.737856 6273 generic.go:356] "Generic (PLEG): container finished" podID="e51213ce-c8a3-4956-ab31-f5ba9de58395" containerID="69f81c20a4c09c8a374132a2bbf50ba375cd6708c0cf913b2a598000760ce7a7" exitCode=0 Nov 02 06:36:54.738102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.737948 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" event={"ID":"e51213ce-c8a3-4956-ab31-f5ba9de58395","Type":"ContainerDied","Data":"69f81c20a4c09c8a374132a2bbf50ba375cd6708c0cf913b2a598000760ce7a7"} Nov 02 06:36:54.922618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.922579 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:54.923110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.923092 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:54.923841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:54.923820 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:36:55.551246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:55.551215 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:55.551506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:55.551483 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:55.552238 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:55.552217 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:36:55.928855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:55.928827 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:56.011878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.011850 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\") pod \"e51213ce-c8a3-4956-ab31-f5ba9de58395\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " Nov 02 06:36:56.012085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.011887 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\" (UniqueName: \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\") pod \"e51213ce-c8a3-4956-ab31-f5ba9de58395\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " Nov 02 06:36:56.012085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.011916 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\") pod \"e51213ce-c8a3-4956-ab31-f5ba9de58395\" (UID: \"e51213ce-c8a3-4956-ab31-f5ba9de58395\") " Nov 02 06:36:56.012319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.012296 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca" pod "e51213ce-c8a3-4956-ab31-f5ba9de58395" (UID: "e51213ce-c8a3-4956-ab31-f5ba9de58395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:56.012465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.012451 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:56.014081 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.014056 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca" pod "e51213ce-c8a3-4956-ab31-f5ba9de58395" (UID: "e51213ce-c8a3-4956-ab31-f5ba9de58395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:56.014197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.014094 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf" pod "e51213ce-c8a3-4956-ab31-f5ba9de58395" (UID: "e51213ce-c8a3-4956-ab31-f5ba9de58395"). InnerVolumeSpecName "kube-api-access-c2gpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:56.114159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.114070 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51213ce-c8a3-4956-ab31-f5ba9de58395-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:56.114159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.114105 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gpf\" (UniqueName: \"kubernetes.io/projected/e51213ce-c8a3-4956-ab31-f5ba9de58395-kube-api-access-c2gpf\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:56.751043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.751015 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" Nov 02 06:36:56.751253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.751035 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr" event={"ID":"e51213ce-c8a3-4956-ab31-f5ba9de58395","Type":"ContainerDied","Data":"9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df"} Nov 02 06:36:56.751253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:36:56.751089 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e15cb555d62ae5bf6c743d28686caf7acbbe502d28f56f5bcda3f2e9b5724df" Nov 02 06:37:00.565884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.565845 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb"] Nov 02 06:37:00.566521 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.566503 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e51213ce-c8a3-4956-ab31-f5ba9de58395" containerName="revoke-keys" Nov 02 06:37:00.566565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.566522 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51213ce-c8a3-4956-ab31-f5ba9de58395" containerName="revoke-keys" Nov 02 06:37:00.566631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.566622 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51213ce-c8a3-4956-ab31-f5ba9de58395" containerName="revoke-keys" Nov 02 06:37:00.570237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.570217 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.623015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.622949 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb"] Nov 02 06:37:00.665863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.665837 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:00.666028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.665937 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:00.669612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.669586 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvzg\" (UniqueName: \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.669750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.669654 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.669831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.669811 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.770932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.770902 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.771104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.770982 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.771196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.771179 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvzg\" (UniqueName: \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.771710 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.771687 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.773297 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.773275 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.825399 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.825337 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvzg\" (UniqueName: \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\") pod \"maas-api-revoke-sub-30c4624a-kfrgb\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:00.883314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:00.883293 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:01.256050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:01.256015 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb"] Nov 02 06:37:01.261858 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:01.261828 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31131fd_d4fb_4e6e_b85a_17936dae1d7a.slice/crio-7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e WatchSource:0}: Error finding container 7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e: Status 404 returned error can't find the container with id 7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e Nov 02 06:37:01.794630 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:01.794585 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" event={"ID":"e31131fd-d4fb-4e6e-b85a-17936dae1d7a","Type":"ContainerStarted","Data":"a8f6b0855e4a19d19a4eb250e6926a12860612b7f96984a82350b1cad3ed68cb"} Nov 02 06:37:01.795134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:01.794674 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" event={"ID":"e31131fd-d4fb-4e6e-b85a-17936dae1d7a","Type":"ContainerStarted","Data":"7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e"} Nov 02 06:37:01.905370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:01.905322 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" podStartSLOduration=1.905308343 podStartE2EDuration="1.905308343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:01.902425498 +0000 UTC m=+1175.875760185" watchObservedRunningTime="2026-11-02 06:37:01.905308343 +0000 UTC m=+1175.878643026" Nov 02 06:37:02.801697 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:02.801659 6273 generic.go:356] "Generic (PLEG): container finished" podID="e31131fd-d4fb-4e6e-b85a-17936dae1d7a" containerID="a8f6b0855e4a19d19a4eb250e6926a12860612b7f96984a82350b1cad3ed68cb" exitCode=0 Nov 02 06:37:02.802179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:02.801707 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" event={"ID":"e31131fd-d4fb-4e6e-b85a-17936dae1d7a","Type":"ContainerDied","Data":"a8f6b0855e4a19d19a4eb250e6926a12860612b7f96984a82350b1cad3ed68cb"} Nov 02 06:37:03.953762 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:03.953734 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:04.109878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.109792 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\" (UniqueName: \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\") pod \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " Nov 02 06:37:04.109878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.109848 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\") pod \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " Nov 02 06:37:04.110147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.109959 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\") pod \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\" (UID: \"e31131fd-d4fb-4e6e-b85a-17936dae1d7a\") " Nov 02 06:37:04.110378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.110348 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca" pod "e31131fd-d4fb-4e6e-b85a-17936dae1d7a" (UID: "e31131fd-d4fb-4e6e-b85a-17936dae1d7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:04.110894 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.110875 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:04.112020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.111971 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca" pod "e31131fd-d4fb-4e6e-b85a-17936dae1d7a" (UID: "e31131fd-d4fb-4e6e-b85a-17936dae1d7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:04.112168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.112151 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg" pod "e31131fd-d4fb-4e6e-b85a-17936dae1d7a" (UID: "e31131fd-d4fb-4e6e-b85a-17936dae1d7a"). InnerVolumeSpecName "kube-api-access-5dvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:04.212628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.212518 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvzg\" (UniqueName: \"kubernetes.io/projected/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-kube-api-access-5dvzg\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:04.212628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.212558 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31131fd-d4fb-4e6e-b85a-17936dae1d7a-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:04.815163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.815135 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" Nov 02 06:37:04.815329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.815171 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb" event={"ID":"e31131fd-d4fb-4e6e-b85a-17936dae1d7a","Type":"ContainerDied","Data":"7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e"} Nov 02 06:37:04.815329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:04.815215 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b85f6fb197ce14e1bf5800acc2492e896ba4fcdc4acb84ecfd9b2f7f32d885e" Nov 02 06:37:05.100548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.100462 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd"] Nov 02 06:37:05.101313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.101291 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e31131fd-d4fb-4e6e-b85a-17936dae1d7a" containerName="revoke-keys" Nov 02 06:37:05.101422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.101313 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31131fd-d4fb-4e6e-b85a-17936dae1d7a" containerName="revoke-keys" Nov 02 06:37:05.101480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.101456 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31131fd-d4fb-4e6e-b85a-17936dae1d7a" containerName="revoke-keys" Nov 02 06:37:05.104867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.104848 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.123848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.123819 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.123949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.123871 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.124003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.123960 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whw9g\" (UniqueName: \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.155576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.155542 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:05.155701 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.155666 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:05.158416 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.158392 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd"] Nov 02 06:37:05.224809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.224776 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.224974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.224835 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.224974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.224886 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whw9g\" (UniqueName: \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.225458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.225424 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.227147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.227123 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.278566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.278535 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whw9g\" (UniqueName: \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\") pod \"maas-api-revoke-sub-9eacec68-9dkrd\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.418610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.418577 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:05.795620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.795536 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd"] Nov 02 06:37:05.801355 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:05.801327 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b27640_3773_456b_a5e6_e2c522b3302e.slice/crio-8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10 WatchSource:0}: Error finding container 8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10: Status 404 returned error can't find the container with id 8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10 Nov 02 06:37:05.823431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:05.823405 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" event={"ID":"a3b27640-3773-456b-a5e6-e2c522b3302e","Type":"ContainerStarted","Data":"8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10"} Nov 02 06:37:06.046881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:06.046798 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k"] Nov 02 06:37:06.054751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:06.054724 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad2add-bfz5k"] Nov 02 06:37:06.186750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:06.186712 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358463fc-e677-463e-9a1f-9d2a48ad96a9" path="/var/lib/kubelet/pods/358463fc-e677-463e-9a1f-9d2a48ad96a9/volumes" Nov 02 06:37:06.833806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:06.833770 6273 generic.go:356] "Generic (PLEG): container finished" podID="a3b27640-3773-456b-a5e6-e2c522b3302e" containerID="60a4ffda0c8d165918bad4fa2d3f008425c23e2e03c6b79ac0507bc6039d0dff" exitCode=0 Nov 02 06:37:06.833931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:06.833894 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" event={"ID":"a3b27640-3773-456b-a5e6-e2c522b3302e","Type":"ContainerDied","Data":"60a4ffda0c8d165918bad4fa2d3f008425c23e2e03c6b79ac0507bc6039d0dff"} Nov 02 06:37:08.336061 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.319146 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.369895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.369836 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.745008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.724617 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.745008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.737080 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.783970 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.782823 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.906467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:08.906153 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:09.016583 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.013930 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.072022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.069142 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" event={"ID":"a3b27640-3773-456b-a5e6-e2c522b3302e","Type":"ContainerDied","Data":"8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10"} Nov 02 06:37:09.072022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.069204 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8235fdb624864d483f25410415578fa7c895b058736dece8beef34d1660afb10" Nov 02 06:37:09.072022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.069281 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd" Nov 02 06:37:09.089023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.083734 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\") pod \"a3b27640-3773-456b-a5e6-e2c522b3302e\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " Nov 02 06:37:09.089023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.083816 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\") pod \"a3b27640-3773-456b-a5e6-e2c522b3302e\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " Nov 02 06:37:09.089023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.083867 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\" (UniqueName: \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\") pod \"a3b27640-3773-456b-a5e6-e2c522b3302e\" (UID: \"a3b27640-3773-456b-a5e6-e2c522b3302e\") " Nov 02 06:37:09.104049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.090323 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca" pod "a3b27640-3773-456b-a5e6-e2c522b3302e" (UID: "a3b27640-3773-456b-a5e6-e2c522b3302e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:09.104049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.090390 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 06:37:09.132215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.132143 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g" pod "a3b27640-3773-456b-a5e6-e2c522b3302e" (UID: "a3b27640-3773-456b-a5e6-e2c522b3302e"). InnerVolumeSpecName "kube-api-access-whw9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:09.198023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.197068 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:09.198023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.197110 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whw9g\" (UniqueName: \"kubernetes.io/projected/a3b27640-3773-456b-a5e6-e2c522b3302e-kube-api-access-whw9g\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:09.222085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.221199 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca" pod "a3b27640-3773-456b-a5e6-e2c522b3302e" (UID: "a3b27640-3773-456b-a5e6-e2c522b3302e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:09.241105 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:37:09.241059 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:37:09.301007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:09.300965 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b27640-3773-456b-a5e6-e2c522b3302e-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:10.767465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.767429 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b"] Nov 02 06:37:10.768035 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.768016 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a3b27640-3773-456b-a5e6-e2c522b3302e" containerName="revoke-keys" Nov 02 06:37:10.768035 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.768032 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b27640-3773-456b-a5e6-e2c522b3302e" containerName="revoke-keys" Nov 02 06:37:10.768232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.768146 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b27640-3773-456b-a5e6-e2c522b3302e" containerName="revoke-keys" Nov 02 06:37:10.771456 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.771438 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.818305 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.818282 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.818420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.818312 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.818471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.818424 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp42\" (UniqueName: \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.869043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.868980 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:10.869202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.869058 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:10.871933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.871908 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b"] Nov 02 06:37:10.919537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.919508 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.919655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.919540 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.919694 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.919660 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp42\" (UniqueName: \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.920125 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.920106 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.921867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.921846 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:10.973199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:10.973169 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp42\" (UniqueName: \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\") pod \"maas-api-revoke-sub-cd3c6760-6754b\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:11.086838 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:11.084687 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:11.460870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:11.460836 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b"] Nov 02 06:37:11.466628 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:11.466599 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb615569_3a3a_4d81_aeaa_7f27b0d3043a.slice/crio-1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a WatchSource:0}: Error finding container 1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a: Status 404 returned error can't find the container with id 1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a Nov 02 06:37:12.099155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:12.099118 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" event={"ID":"db615569-3a3a-4d81-aeaa-7f27b0d3043a","Type":"ContainerStarted","Data":"7c233b460ec234c64a9d08d877962e25abf47df0caf957aed4dac28a8ddc6dc1"} Nov 02 06:37:12.099562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:12.099184 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" event={"ID":"db615569-3a3a-4d81-aeaa-7f27b0d3043a","Type":"ContainerStarted","Data":"1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a"} Nov 02 06:37:12.207835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:12.207783 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" podStartSLOduration=2.207771064 podStartE2EDuration="2.207771064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:12.205523421 +0000 UTC m=+1186.178858105" watchObservedRunningTime="2026-11-02 06:37:12.207771064 +0000 UTC m=+1186.181105747" Nov 02 06:37:13.108170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:13.108126 6273 generic.go:356] "Generic (PLEG): container finished" podID="db615569-3a3a-4d81-aeaa-7f27b0d3043a" containerID="7c233b460ec234c64a9d08d877962e25abf47df0caf957aed4dac28a8ddc6dc1" exitCode=0 Nov 02 06:37:13.108561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:13.108173 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" event={"ID":"db615569-3a3a-4d81-aeaa-7f27b0d3043a","Type":"ContainerDied","Data":"7c233b460ec234c64a9d08d877962e25abf47df0caf957aed4dac28a8ddc6dc1"} Nov 02 06:37:14.263134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.263109 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:14.361132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.361098 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\") pod \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " Nov 02 06:37:14.361348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.361162 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\" (UniqueName: \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\") pod \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " Nov 02 06:37:14.361522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.361475 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\") pod \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\" (UID: \"db615569-3a3a-4d81-aeaa-7f27b0d3043a\") " Nov 02 06:37:14.362752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.362245 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca" pod "db615569-3a3a-4d81-aeaa-7f27b0d3043a" (UID: "db615569-3a3a-4d81-aeaa-7f27b0d3043a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:14.364786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.364691 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca" pod "db615569-3a3a-4d81-aeaa-7f27b0d3043a" (UID: "db615569-3a3a-4d81-aeaa-7f27b0d3043a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:14.365549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.365513 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42" pod "db615569-3a3a-4d81-aeaa-7f27b0d3043a" (UID: "db615569-3a3a-4d81-aeaa-7f27b0d3043a"). InnerVolumeSpecName "kube-api-access-khp42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:14.463667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.463635 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:14.463667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.463665 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db615569-3a3a-4d81-aeaa-7f27b0d3043a-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:14.463863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:14.463676 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khp42\" (UniqueName: \"kubernetes.io/projected/db615569-3a3a-4d81-aeaa-7f27b0d3043a-kube-api-access-khp42\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:15.121495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:15.121468 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" Nov 02 06:37:15.121666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:15.121467 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b" event={"ID":"db615569-3a3a-4d81-aeaa-7f27b0d3043a","Type":"ContainerDied","Data":"1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a"} Nov 02 06:37:15.121666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:15.121617 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1141fe502957bee3a292acf0fed1ffd01f89edd8adb83393f7400f7bd9bfcf8a" Nov 02 06:37:22.049279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:22.049244 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq"] Nov 02 06:37:22.099122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:22.097660 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ade36c2-28tmq"] Nov 02 06:37:22.180507 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:22.180466 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64ff033-ac80-40e9-a258-540c3c923d6d" path="/var/lib/kubelet/pods/e64ff033-ac80-40e9-a258-540c3c923d6d/volumes" Nov 02 06:37:23.826197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.826161 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99"] Nov 02 06:37:23.826802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.826787 6273 cpu_manager.go:414] "containerMap: removing container" podUID="db615569-3a3a-4d81-aeaa-7f27b0d3043a" containerName="revoke-keys" Nov 02 06:37:23.826853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.826804 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db615569-3a3a-4d81-aeaa-7f27b0d3043a" containerName="revoke-keys" Nov 02 06:37:23.826920 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.826910 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="db615569-3a3a-4d81-aeaa-7f27b0d3043a" containerName="revoke-keys" Nov 02 06:37:23.831403 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.831385 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:23.884633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.884599 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99"] Nov 02 06:37:23.926901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.926861 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:23.927080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.926930 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:23.976131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.976098 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:23.976304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.976260 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:23.976392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:23.976376 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwrq\" (UniqueName: \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.078072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.077975 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwrq\" (UniqueName: \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.078072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.078058 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.078280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.078184 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.078866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.078835 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.080884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.080857 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.131727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.131693 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwrq\" (UniqueName: \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\") pod \"maas-api-revoke-sub-8ac5fd2d-svz99\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.145217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.145187 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:24.518738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:24.518697 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99"] Nov 02 06:37:24.525584 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:24.525550 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f2a2a_9a48_4f92_8236_6bafdc8e6456.slice/crio-0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd WatchSource:0}: Error finding container 0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd: Status 404 returned error can't find the container with id 0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd Nov 02 06:37:25.190112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:25.190082 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" event={"ID":"647f2a2a-9a48-4f92-8236-6bafdc8e6456","Type":"ContainerStarted","Data":"e9f511f936512d6e81b7eb3fb13335b455aaec761e8f9450114037936200cdbf"} Nov 02 06:37:25.190481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:25.190129 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" event={"ID":"647f2a2a-9a48-4f92-8236-6bafdc8e6456","Type":"ContainerStarted","Data":"0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd"} Nov 02 06:37:25.298674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:25.298628 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" podStartSLOduration=2.298614852 podStartE2EDuration="2.298614852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:25.297090177 +0000 UTC m=+1199.270424861" watchObservedRunningTime="2026-11-02 06:37:25.298614852 +0000 UTC m=+1199.271949582" Nov 02 06:37:26.198413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:26.198381 6273 generic.go:356] "Generic (PLEG): container finished" podID="647f2a2a-9a48-4f92-8236-6bafdc8e6456" containerID="e9f511f936512d6e81b7eb3fb13335b455aaec761e8f9450114037936200cdbf" exitCode=0 Nov 02 06:37:26.198838 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:26.198461 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" event={"ID":"647f2a2a-9a48-4f92-8236-6bafdc8e6456","Type":"ContainerDied","Data":"e9f511f936512d6e81b7eb3fb13335b455aaec761e8f9450114037936200cdbf"} Nov 02 06:37:27.365963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.365936 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:27.523081 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.522984 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\") pod \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " Nov 02 06:37:27.523224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.523141 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\" (UniqueName: \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\") pod \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " Nov 02 06:37:27.523224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.523199 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\") pod \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\" (UID: \"647f2a2a-9a48-4f92-8236-6bafdc8e6456\") " Nov 02 06:37:27.523296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.523253 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca" pod "647f2a2a-9a48-4f92-8236-6bafdc8e6456" (UID: "647f2a2a-9a48-4f92-8236-6bafdc8e6456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:27.523633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.523616 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:27.525244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.525218 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq" pod "647f2a2a-9a48-4f92-8236-6bafdc8e6456" (UID: "647f2a2a-9a48-4f92-8236-6bafdc8e6456"). InnerVolumeSpecName "kube-api-access-6bwrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:27.525366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.525263 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca" pod "647f2a2a-9a48-4f92-8236-6bafdc8e6456" (UID: "647f2a2a-9a48-4f92-8236-6bafdc8e6456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:27.625230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.625194 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f2a2a-9a48-4f92-8236-6bafdc8e6456-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:27.625230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:27.625224 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwrq\" (UniqueName: \"kubernetes.io/projected/647f2a2a-9a48-4f92-8236-6bafdc8e6456-kube-api-access-6bwrq\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:28.212412 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.212364 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" Nov 02 06:37:28.212412 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.212390 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99" event={"ID":"647f2a2a-9a48-4f92-8236-6bafdc8e6456","Type":"ContainerDied","Data":"0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd"} Nov 02 06:37:28.212650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.212442 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bac505c7787ff89d46658e28c66d49d65567ef913bdfa20ae6a577522b5b2bd" Nov 02 06:37:28.483513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.483434 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs"] Nov 02 06:37:28.483983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.483970 6273 cpu_manager.go:414] "containerMap: removing container" podUID="647f2a2a-9a48-4f92-8236-6bafdc8e6456" containerName="revoke-keys" Nov 02 06:37:28.484057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.484010 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f2a2a-9a48-4f92-8236-6bafdc8e6456" containerName="revoke-keys" Nov 02 06:37:28.484142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.484133 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f2a2a-9a48-4f92-8236-6bafdc8e6456" containerName="revoke-keys" Nov 02 06:37:28.488226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.488204 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.538904 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.538552 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:28.539094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.538604 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:28.542429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.542404 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs"] Nov 02 06:37:28.639051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.638980 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.639209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.639097 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.639209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.639149 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhz8\" (UniqueName: \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.740978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.740891 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.740978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.740949 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.741235 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.741036 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhz8\" (UniqueName: \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.741527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.741501 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.743313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.743290 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.795832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.795804 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhz8\" (UniqueName: \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\") pod \"maas-api-revoke-sub-4ac85fe0-qsqfs\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:28.802795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:28.802778 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:29.179425 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:29.179391 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs"] Nov 02 06:37:29.185226 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:29.185199 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a8fe3b_db2c_450f_91b2_43c3da75878c.slice/crio-6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f WatchSource:0}: Error finding container 6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f: Status 404 returned error can't find the container with id 6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f Nov 02 06:37:29.223772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:29.223740 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" event={"ID":"98a8fe3b-db2c-450f-91b2-43c3da75878c","Type":"ContainerStarted","Data":"6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f"} Nov 02 06:37:30.232234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:30.232195 6273 generic.go:356] "Generic (PLEG): container finished" podID="98a8fe3b-db2c-450f-91b2-43c3da75878c" containerID="f4c93cf56aed6cb150c4f88fb85abdecbebbd97990687b1e2ed31ee119fda52e" exitCode=0 Nov 02 06:37:30.232555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:30.232276 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" event={"ID":"98a8fe3b-db2c-450f-91b2-43c3da75878c","Type":"ContainerDied","Data":"f4c93cf56aed6cb150c4f88fb85abdecbebbd97990687b1e2ed31ee119fda52e"} Nov 02 06:37:31.387753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.387728 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:31.572631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.572551 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\") pod \"98a8fe3b-db2c-450f-91b2-43c3da75878c\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " Nov 02 06:37:31.572820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.572638 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\") pod \"98a8fe3b-db2c-450f-91b2-43c3da75878c\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " Nov 02 06:37:31.572820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.572659 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\" (UniqueName: \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\") pod \"98a8fe3b-db2c-450f-91b2-43c3da75878c\" (UID: \"98a8fe3b-db2c-450f-91b2-43c3da75878c\") " Nov 02 06:37:31.573006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.572962 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca" pod "98a8fe3b-db2c-450f-91b2-43c3da75878c" (UID: "98a8fe3b-db2c-450f-91b2-43c3da75878c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:31.573240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.573221 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:31.574662 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.574640 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca" pod "98a8fe3b-db2c-450f-91b2-43c3da75878c" (UID: "98a8fe3b-db2c-450f-91b2-43c3da75878c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:31.574818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.574800 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8" pod "98a8fe3b-db2c-450f-91b2-43c3da75878c" (UID: "98a8fe3b-db2c-450f-91b2-43c3da75878c"). InnerVolumeSpecName "kube-api-access-xbhz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:31.674615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.674586 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a8fe3b-db2c-450f-91b2-43c3da75878c-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:31.674615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:31.674614 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhz8\" (UniqueName: \"kubernetes.io/projected/98a8fe3b-db2c-450f-91b2-43c3da75878c-kube-api-access-xbhz8\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:32.245867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.245838 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" Nov 02 06:37:32.245867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.245837 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs" event={"ID":"98a8fe3b-db2c-450f-91b2-43c3da75878c","Type":"ContainerDied","Data":"6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f"} Nov 02 06:37:32.246119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.245899 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6222c6473ddb764ffc2444b9bf8d17a1cb7d158d751d88fdb779367e03b4f97f" Nov 02 06:37:32.354924 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.354893 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:37:32.355483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.355467 6273 cpu_manager.go:414] "containerMap: removing container" podUID="98a8fe3b-db2c-450f-91b2-43c3da75878c" containerName="revoke-keys" Nov 02 06:37:32.355538 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.355485 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a8fe3b-db2c-450f-91b2-43c3da75878c" containerName="revoke-keys" Nov 02 06:37:32.355617 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.355608 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a8fe3b-db2c-450f-91b2-43c3da75878c" containerName="revoke-keys" Nov 02 06:37:32.359313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.359291 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.418057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.418024 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:37:32.484757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.484670 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.484757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.484719 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.484947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.484867 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.484947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.484938 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.485059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.485024 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.485059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.485052 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.485141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.485077 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82vc\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.485243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.485223 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.485335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.485318 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.586804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.586768 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.586808 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.586875 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.586929 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.586972 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587030 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587056 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82vc\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587236 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587254 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587314 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587341 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587422 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.587565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587527 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.588015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.587978 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.589338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.589318 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.589433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.589403 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.642916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.642881 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.643036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.643002 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82vc\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.673364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.673336 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:32.794835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.794803 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:37:32.800685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.800665 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.854375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.854348 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:37:32.994131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994041 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994179 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994209 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994241 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nbd\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994369 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994422 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994470 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994500 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:32.994569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:32.994556 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096034 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096153 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096207 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096305 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096343 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096395 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nbd\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096476 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096543 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096591 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.096883 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.096837 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.097665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.097442 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.097665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.097538 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.098536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.097797 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.099277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.099255 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.100074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.100055 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.100702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.100681 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.114895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.114839 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:37:33.120520 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:33.120496 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode951fc98_7e26_463d_9eb8_5fe78830aad2.slice/crio-266b35a793c6b33242da4dcfaa47333f18dc289a3930ddf32a2f7536250ca643 WatchSource:0}: Error finding container 266b35a793c6b33242da4dcfaa47333f18dc289a3930ddf32a2f7536250ca643: Status 404 returned error can't find the container with id 266b35a793c6b33242da4dcfaa47333f18dc289a3930ddf32a2f7536250ca643 Nov 02 06:37:33.123337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.123302 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.123438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.123373 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.123438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.123402 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.150454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.150429 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.150566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.150525 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nbd\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.195416 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.195384 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx"] Nov 02 06:37:33.199395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.199375 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.251411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.251318 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:33.251411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.251370 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:33.255300 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.255270 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx"] Nov 02 06:37:33.256589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.256557 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" event={"ID":"e951fc98-7e26-463d-9eb8-5fe78830aad2","Type":"ContainerStarted","Data":"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687"} Nov 02 06:37:33.256739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.256613 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" event={"ID":"e951fc98-7e26-463d-9eb8-5fe78830aad2","Type":"ContainerStarted","Data":"266b35a793c6b33242da4dcfaa47333f18dc289a3930ddf32a2f7536250ca643"} Nov 02 06:37:33.299634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.299604 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.299799 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.299765 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnm5\" (UniqueName: \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.299872 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.299855 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.401513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.401466 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.401715 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.401664 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnm5\" (UniqueName: \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.401789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.401774 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.402310 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.402284 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.404266 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.404247 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.416383 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.416342 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:33.421001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.420940 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podStartSLOduration=1.420927354 podStartE2EDuration="1.420927354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:33.41681259 +0000 UTC m=+1207.390147300" watchObservedRunningTime="2026-11-02 06:37:33.420927354 +0000 UTC m=+1207.394262037" Nov 02 06:37:33.455848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.455811 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnm5\" (UniqueName: \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\") pod \"maas-api-revoke-sub-fb239dfc-m59kx\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.517206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.513962 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:33.673957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.673927 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:33.854147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.854073 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:37:33.899244 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:33.899216 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4854653d_ecd4_49dc_8924_67b9c1f96241.slice/crio-2594b0f6557f79b55cc4fb99cc4c39c4e67a62116e4c385748455959e8eb1bce WatchSource:0}: Error finding container 2594b0f6557f79b55cc4fb99cc4c39c4e67a62116e4c385748455959e8eb1bce: Status 404 returned error can't find the container with id 2594b0f6557f79b55cc4fb99cc4c39c4e67a62116e4c385748455959e8eb1bce Nov 02 06:37:33.901895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.901861 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.902025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.901930 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.902025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.901972 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.906030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:33.905971 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx"] Nov 02 06:37:33.913670 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:33.913647 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b44e68c_1fac_4bfa_bab5_e11c49e140f6.slice/crio-fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f WatchSource:0}: Error finding container fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f: Status 404 returned error can't find the container with id fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f Nov 02 06:37:34.287422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.287389 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" event={"ID":"4854653d-ecd4-49dc-8924-67b9c1f96241","Type":"ContainerStarted","Data":"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7"} Nov 02 06:37:34.287635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.287452 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" event={"ID":"4854653d-ecd4-49dc-8924-67b9c1f96241","Type":"ContainerStarted","Data":"2594b0f6557f79b55cc4fb99cc4c39c4e67a62116e4c385748455959e8eb1bce"} Nov 02 06:37:34.290218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.290169 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" event={"ID":"6b44e68c-1fac-4bfa-bab5-e11c49e140f6","Type":"ContainerStarted","Data":"fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f"} Nov 02 06:37:34.399304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.399253 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podStartSLOduration=2.399235986 podStartE2EDuration="2.399235986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:34.398458339 +0000 UTC m=+1208.371793023" watchObservedRunningTime="2026-11-02 06:37:34.399235986 +0000 UTC m=+1208.372570672" Nov 02 06:37:34.417332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.417304 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:34.418928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.418904 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.419065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.418953 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:34.677166 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.677128 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.677762 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:34.677730 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:35.307622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.307538 6273 generic.go:356] "Generic (PLEG): container finished" podID="6b44e68c-1fac-4bfa-bab5-e11c49e140f6" containerID="fc643e3b45c76b840a8704a0b98da76f9848e649b431624990961393a2fc5f82" exitCode=0 Nov 02 06:37:35.307866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.307615 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" event={"ID":"6b44e68c-1fac-4bfa-bab5-e11c49e140f6","Type":"ContainerDied","Data":"fc643e3b45c76b840a8704a0b98da76f9848e649b431624990961393a2fc5f82"} Nov 02 06:37:35.416650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.416606 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.416851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.416664 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:35.690015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.676679 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.690015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:35.676757 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:36.416619 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.416581 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.416793 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.416652 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:36.609053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.608247 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:36.647842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.647817 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\") pod \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " Nov 02 06:37:36.647947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.647881 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\" (UniqueName: \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\") pod \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " Nov 02 06:37:36.648064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.648040 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\") pod \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\" (UID: \"6b44e68c-1fac-4bfa-bab5-e11c49e140f6\") " Nov 02 06:37:36.660010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.656167 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca" pod "6b44e68c-1fac-4bfa-bab5-e11c49e140f6" (UID: "6b44e68c-1fac-4bfa-bab5-e11c49e140f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:36.697008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.678594 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.697008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.678659 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:36.704417 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.704371 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5" pod "6b44e68c-1fac-4bfa-bab5-e11c49e140f6" (UID: "6b44e68c-1fac-4bfa-bab5-e11c49e140f6"). InnerVolumeSpecName "kube-api-access-sdnm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:36.707876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.707844 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca" pod "6b44e68c-1fac-4bfa-bab5-e11c49e140f6" (UID: "6b44e68c-1fac-4bfa-bab5-e11c49e140f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:36.776012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.753094 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:36.776315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.776271 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:36.776457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:36.776443 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnm5\" (UniqueName: \"kubernetes.io/projected/6b44e68c-1fac-4bfa-bab5-e11c49e140f6-kube-api-access-sdnm5\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:37.323695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.323669 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" Nov 02 06:37:37.323962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.323706 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx" event={"ID":"6b44e68c-1fac-4bfa-bab5-e11c49e140f6","Type":"ContainerDied","Data":"fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f"} Nov 02 06:37:37.324041 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.324009 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc15428dd6f179bf56808c1cb93ff380f44ace363e96e52ee071ea307ff52f6f" Nov 02 06:37:37.416954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.416914 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.417160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.417016 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:37.681114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.681078 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.681410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:37.681377 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:38.069140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.069041 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm"] Nov 02 06:37:38.078011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.077608 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-992efb99-prxtm"] Nov 02 06:37:38.204809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.204764 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafebcaf-aae2-43da-aac0-990cd46c9ecb" path="/var/lib/kubelet/pods/fafebcaf-aae2-43da-aac0-990cd46c9ecb/volumes" Nov 02 06:37:38.416863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.416822 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.417053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.416892 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:38.613106 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:37:38.612760 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:37:38.708140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.707136 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.708140 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:38.707194 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:39.407603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:39.407562 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:37:39.417466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:39.417434 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.417635 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:39.417488 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:39.681079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:39.680965 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.681079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:39.681037 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:40.423421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:40.423058 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.423421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:40.423143 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:40.687110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:40.686631 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.711013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:40.697077 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 06:37:41.417458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:41.417416 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 06:37:41.417744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:41.417714 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 06:37:42.476368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.476330 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:42.478485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.478462 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:42.486240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.486217 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:37:42.519131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.519074 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:37:42.641310 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.641265 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:37:42.669155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.669104 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.687015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.683069 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.789979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:42.789885 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.091191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.091090 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.172267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.172221 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:37:43.268014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.264620 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.303014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.299079 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.411076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.385382 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.519012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.487884 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.519012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.487944 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.723068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.720377 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 02 06:37:43.723068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.721891 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.724661 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.724633 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:37:43.793013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.731126 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:37:43.805014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.801106 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:37:43.813306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.733624 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:37:43.813843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.813790 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:37:43.814429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.790883 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 06:37:43.814922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.733594 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.815393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.815370 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:37:43.815565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.815551 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:37:43.915901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.915859 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.928019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:43.916510 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.372825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.372756 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.398743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.398705 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:37:44.567749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.455697 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.607050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.569190 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.672877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.672831 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:37:44.903049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.902618 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:37:44.908012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.904868 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:44.908012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.904932 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.940067 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:44.913234 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:37:45.135000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.077090 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.135000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.077980 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 02 06:37:45.278238 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.190803 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:37:45.291311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.196139 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 06:37:45.317011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.196169 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 06:37:45.465460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.210523 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:37:45.489786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.317780 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:37:45.490113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.490090 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:37:45.498216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.498188 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:37:45.498435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.498419 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:37:45.498585 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.498572 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:37:45.498703 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.498689 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:37:45.534070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.398396 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:37:45.545429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.525771 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6b44e68c-1fac-4bfa-bab5-e11c49e140f6" containerName="revoke-keys" Nov 02 06:37:45.545647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.545626 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b44e68c-1fac-4bfa-bab5-e11c49e140f6" containerName="revoke-keys" Nov 02 06:37:45.546043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.546022 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b44e68c-1fac-4bfa-bab5-e11c49e140f6" containerName="revoke-keys" Nov 02 06:37:45.608610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.607970 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.642563 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.642614 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.642687 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.642719 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.642774 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.643099 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.649532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.648173 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.662149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.662124 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:37:45.709025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.695910 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15867003-openshift-default-dockercfg-92xfd\"" Nov 02 06:37:45.709025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.701034 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:37:45.709025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.701121 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:37:45.709025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.704055 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:37:45.709025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.704629 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:37:45.712054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.712034 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:37:45.724699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.724678 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:37:45.790586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.790493 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.790586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.790572 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.790844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.790630 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.790844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.790818 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgh9x\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.790971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.790935 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.791177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.791141 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.791274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.791191 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.791274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.791244 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.791391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.791280 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.893571 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.893530 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.893760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.893677 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.893760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.893726 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.893885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.893849 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgh9x\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.893983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.893959 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.894181 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.894155 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.894447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.894425 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.894601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.894583 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895109 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.895070 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.895289 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.895739 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.894976 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895843 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.895578 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.895965 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.894690 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.896655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.896631 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.896754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.896727 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.947408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.947371 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:45.947588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.947431 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.949741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.949710 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgh9x\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:45.950095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:45.950064 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\") pod \"e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:46.010397 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.010365 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:46.060533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.060428 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:46.061089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.061053 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:46.063866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.063842 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:37:46.454233 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.454196 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:37:46.458674 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:46.458646 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dec9a4_b77d_4614_a21a_db0216a0bf55.slice/crio-e82bddcc7ddcb595621fea875bf2d9a632f3d1c8c1f81507ee3c699b4e2c0f8e WatchSource:0}: Error finding container e82bddcc7ddcb595621fea875bf2d9a632f3d1c8c1f81507ee3c699b4e2c0f8e: Status 404 returned error can't find the container with id e82bddcc7ddcb595621fea875bf2d9a632f3d1c8c1f81507ee3c699b4e2c0f8e Nov 02 06:37:46.461289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.461258 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:46.461377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.461325 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:46.461377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.461356 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:46.613725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.613683 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerStarted","Data":"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084"} Nov 02 06:37:46.614206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:46.613747 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerStarted","Data":"e82bddcc7ddcb595621fea875bf2d9a632f3d1c8c1f81507ee3c699b4e2c0f8e"} Nov 02 06:37:47.011048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.010982 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:47.024508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.024477 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:47.377312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.377219 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podStartSLOduration=3.377205144 podStartE2EDuration="3.377205144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:47.376470413 +0000 UTC m=+1221.349805098" watchObservedRunningTime="2026-11-02 06:37:47.377205144 +0000 UTC m=+1221.350539828" Nov 02 06:37:47.415664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.415630 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:37:47.422477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.422446 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.489810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.489775 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:37:47.522817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.522783 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.522859 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f5n\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.522959 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523010 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523095 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523145 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523276 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523439 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.523592 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.523482 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.622255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.622215 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:47.623285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.623261 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:37:47.624732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624712 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.624795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624750 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.624857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624838 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.624923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624906 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624948 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.624982 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625099 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625070 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f5n\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625125 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625166 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625227 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625397 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625372 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625387 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.625563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.625532 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.626122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.626098 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.627758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.627699 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.627854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.627829 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.694317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.694280 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f5n\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.696611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.696585 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:47.741732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:47.741705 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:48.137339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.137298 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:37:48.142577 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:48.142551 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d87ffb4_0106_4de1_bc5b_ae552d514f7c.slice/crio-5f3bda28e0e4cd214715cb65fa1761d9ba72bcb559598a38f43cb535269bcc4e WatchSource:0}: Error finding container 5f3bda28e0e4cd214715cb65fa1761d9ba72bcb559598a38f43cb535269bcc4e: Status 404 returned error can't find the container with id 5f3bda28e0e4cd214715cb65fa1761d9ba72bcb559598a38f43cb535269bcc4e Nov 02 06:37:48.145367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.145327 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.145442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.145415 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.145490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.145459 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.630664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.630626 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" event={"ID":"7d87ffb4-0106-4de1-bc5b-ae552d514f7c","Type":"ContainerStarted","Data":"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0"} Nov 02 06:37:48.631122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.630680 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" event={"ID":"7d87ffb4-0106-4de1-bc5b-ae552d514f7c","Type":"ContainerStarted","Data":"5f3bda28e0e4cd214715cb65fa1761d9ba72bcb559598a38f43cb535269bcc4e"} Nov 02 06:37:48.741884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.741851 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:48.762379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:48.762329 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podStartSLOduration=1.762311264 podStartE2EDuration="1.762311264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:48.758264803 +0000 UTC m=+1222.731599509" watchObservedRunningTime="2026-11-02 06:37:48.762311264 +0000 UTC m=+1222.735645951" Nov 02 06:37:49.743581 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:49.743552 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:49.744005 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:49.743601 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:50.044998 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:50.044894 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x"] Nov 02 06:37:50.094303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:50.094262 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e156d040-7fc4x"] Nov 02 06:37:50.181213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:50.181180 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488bf6f1-bea4-4da2-8482-3bbf4cd1930c" path="/var/lib/kubelet/pods/488bf6f1-bea4-4da2-8482-3bbf4cd1930c/volumes" Nov 02 06:37:50.745183 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:50.745147 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:50.745921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:50.745890 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:51.075042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.074914 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:37:51.081513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.081488 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.105863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.105836 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8w9\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.105945 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106026 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106077 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106110 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106138 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106173 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106197 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.106403 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.106238 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.118632 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.118604 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:37:51.178173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.178137 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5eed1bfb-gw-openshift-default-dockercfg-5swfq\"" Nov 02 06:37:51.208168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208132 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208174 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208215 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208370 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8w9\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208513 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208598 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208655 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208660 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208691 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.208784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.208723 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.209309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.209282 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.209917 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.209893 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.215187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.211563 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.215187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.212749 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.215187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.213132 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.215600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.215575 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.293227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.289664 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.293227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.291137 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8w9\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\") pod \"e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.397253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.397161 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:51.742656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.742626 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:51.742815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.742688 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:51.875289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.875247 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:37:51.879900 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:37:51.879875 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f775a69_1941_45d4_ae5e_50638498c8e9.slice/crio-631c93c81c10840fa182f94f878a416eaa817edc32cdcd128925698a6589ea98 WatchSource:0}: Error finding container 631c93c81c10840fa182f94f878a416eaa817edc32cdcd128925698a6589ea98: Status 404 returned error can't find the container with id 631c93c81c10840fa182f94f878a416eaa817edc32cdcd128925698a6589ea98 Nov 02 06:37:51.882535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.882501 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:51.882632 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.882597 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:51.882696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:51.882647 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:52.724278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:52.724234 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerStarted","Data":"2c9fa6a804bea5a58eadaf389b8c662fa3f18263824a84cfa69e3382e72c05d7"} Nov 02 06:37:52.724440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:52.724296 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerStarted","Data":"631c93c81c10840fa182f94f878a416eaa817edc32cdcd128925698a6589ea98"} Nov 02 06:37:52.742450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:52.742413 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:52.742591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:52.742469 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:52.927206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:52.927154 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podStartSLOduration=1.927139033 podStartE2EDuration="1.927139033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:52.888641772 +0000 UTC m=+1226.861976458" watchObservedRunningTime="2026-11-02 06:37:52.927139033 +0000 UTC m=+1226.900473714" Nov 02 06:37:53.397524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.397484 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:53.404457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.404426 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:53.736776 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.736741 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:53.738038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.737873 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:37:53.775057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.743446 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:53.775057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:53.743500 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:54.742394 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:54.742352 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:37:54.742817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:54.742414 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 06:37:55.814527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:55.814487 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:55.815271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:55.814811 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:37:55.815765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:37:55.815731 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:38:01.448660 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.448629 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:38:01.454097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.454075 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.492971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.492944 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:38:01.496921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.496904 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.587459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.587424 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:38:01.592629 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.592603 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:38:01.592793 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.592694 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.636420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636384 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txj8b\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.636586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636471 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.636586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636503 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.636586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636520 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.636586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636551 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.636826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636607 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.636826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636707 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.636935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636844 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.636935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636867 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.636935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636888 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.637118 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.636968 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.637199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637172 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.637312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637219 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.637312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637306 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.637424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637334 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.637424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637398 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.637534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637435 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.637534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.637458 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2k2\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.658262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.651524 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:38:01.658262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.657339 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:38:01.739527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739445 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.739527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739498 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.739711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739572 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.739711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739608 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.739711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739673 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.739711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739705 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.739868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739756 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.739868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739782 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.739868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739829 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739866 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739893 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2k2\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.739949 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms4p\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740041 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txj8b\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740067 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740097 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740125 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740138 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740165 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740194 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740227 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740247 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740287 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740330 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740350 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740350 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.740806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740511 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.740806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740523 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.740806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740555 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.740806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740745 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.741018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740912 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.741018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740943 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.741018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.740974 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.741184 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.741059 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.741184 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.741158 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.741184 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.741066 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.741356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.741295 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.741356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.741334 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.743126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.743100 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.743126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.743115 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.743327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.743173 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.743327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.743224 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.794751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.794720 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.795034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.795011 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txj8b\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.795034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.795021 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2k2\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.795658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.795639 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:01.810602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.810579 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:01.840945 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.840919 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:38:01.842761 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.842731 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.842899 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.842882 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms4p\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.842951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.842937 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.842965 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843078 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843061 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843084 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843214 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843263 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843434 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843318 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.843750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.843725 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.844290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.844269 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.844401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.844381 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.844714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.844693 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.844807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.844785 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.846621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.846597 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.846621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.846609 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.847603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.847583 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.901001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.900926 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms4p\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.903730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.903657 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:38:01.904423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.904396 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.912509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.912486 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:01.945041 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.944980 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945156 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945133 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945237 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945355 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945434 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4294\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945468 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945534 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945660 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945628 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:01.945749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:01.945706 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.046983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.046912 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.046983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.046976 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047089 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047131 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047158 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047216 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047209 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047281 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4294\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047370 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047644 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047853 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.047936 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.047310 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.048053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.048034 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.048203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.048179 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.048459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.048438 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.049553 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.049527 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.051851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.051828 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.068922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.068901 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:02.101668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.101635 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.101873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.101849 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4294\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.164560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.164522 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:02.194419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.194391 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:38:02.203078 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:02.203044 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cac6543_e4b4_4b5f_aa53_3aebb7c8270b.slice/crio-fac7f87863fa570b6e3271168530da03c2686c5e5d28a9fe4e953610c019f039 WatchSource:0}: Error finding container fac7f87863fa570b6e3271168530da03c2686c5e5d28a9fe4e953610c019f039: Status 404 returned error can't find the container with id fac7f87863fa570b6e3271168530da03c2686c5e5d28a9fe4e953610c019f039 Nov 02 06:38:02.206293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.206256 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.206420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.206343 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.206420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.206392 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.315142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.314406 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:38:02.315227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.315119 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.315227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.315201 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.315332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.315240 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.450572 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.450539 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:38:02.456065 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:02.456038 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7073f1_06af_4968_b7d3_c138d240ce9b.slice/crio-641455c9db6d7aa8a50ebd7fe330112c08a1314d0aaa6c779a2982a65ba2a572 WatchSource:0}: Error finding container 641455c9db6d7aa8a50ebd7fe330112c08a1314d0aaa6c779a2982a65ba2a572: Status 404 returned error can't find the container with id 641455c9db6d7aa8a50ebd7fe330112c08a1314d0aaa6c779a2982a65ba2a572 Nov 02 06:38:02.459290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.459246 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.459381 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.459338 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.459423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.459387 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.565131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.565099 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:38:02.566200 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:02.566159 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2454edcd_0643_4d9c_9755_3d23403c47d3.slice/crio-d37ad6ff26c1570b6c5aa58bb9f01789caa12d780ddb90db375dffc68bca2435 WatchSource:0}: Error finding container d37ad6ff26c1570b6c5aa58bb9f01789caa12d780ddb90db375dffc68bca2435: Status 404 returned error can't find the container with id d37ad6ff26c1570b6c5aa58bb9f01789caa12d780ddb90db375dffc68bca2435 Nov 02 06:38:02.570243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.570200 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.570358 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.570292 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.570358 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.570338 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.859531 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" event={"ID":"2454edcd-0643-4d9c-9755-3d23403c47d3","Type":"ContainerStarted","Data":"80040da937316436666df9c8a47f102f936d7a1b3195e6130706776306fed739"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.859592 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" event={"ID":"2454edcd-0643-4d9c-9755-3d23403c47d3","Type":"ContainerStarted","Data":"d37ad6ff26c1570b6c5aa58bb9f01789caa12d780ddb90db375dffc68bca2435"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.862909 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" event={"ID":"24049b0f-974b-40b9-a7dd-5cfc2d137ebf","Type":"ContainerStarted","Data":"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.862961 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" event={"ID":"24049b0f-974b-40b9-a7dd-5cfc2d137ebf","Type":"ContainerStarted","Data":"0399234d7c088df84aa074caf838f6c8aa4ecb8a09497cf330ed9c51d1793daf"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.865169 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" event={"ID":"9d7073f1-06af-4968-b7d3-c138d240ce9b","Type":"ContainerStarted","Data":"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15"} Nov 02 06:38:02.866064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.865215 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" event={"ID":"9d7073f1-06af-4968-b7d3-c138d240ce9b","Type":"ContainerStarted","Data":"641455c9db6d7aa8a50ebd7fe330112c08a1314d0aaa6c779a2982a65ba2a572"} Nov 02 06:38:02.867816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.867253 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" event={"ID":"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b","Type":"ContainerStarted","Data":"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e"} Nov 02 06:38:02.867816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.867298 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" event={"ID":"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b","Type":"ContainerStarted","Data":"fac7f87863fa570b6e3271168530da03c2686c5e5d28a9fe4e953610c019f039"} Nov 02 06:38:02.912907 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.912873 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:02.978009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:02.976307 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podStartSLOduration=1.976287579 podStartE2EDuration="1.976287579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:02.975510018 +0000 UTC m=+1236.948844702" watchObservedRunningTime="2026-11-02 06:38:02.976287579 +0000 UTC m=+1236.949622264" Nov 02 06:38:03.044639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.044604 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:38:03.051518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.051489 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.074924 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.074896 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:03.075247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.075223 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.076911 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.076883 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:03.102013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.101504 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podStartSLOduration=2.101486077 podStartE2EDuration="2.101486077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.101108052 +0000 UTC m=+1237.074442736" watchObservedRunningTime="2026-11-02 06:38:03.101486077 +0000 UTC m=+1237.074820762" Nov 02 06:38:03.166389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.166360 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:03.177661 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.177340 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.177661 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.177419 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346586 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwsx\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346654 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346721 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346811 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346858 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.346904 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.347050 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.347106 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.353240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.347141 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.456386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.456036 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.456386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.456108 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.456386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.456150 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.456386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.456223 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.466309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.466265 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.467162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.467138 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.476340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.476311 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:38:03.500009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.490509 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.501717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.501687 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.502795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.456269 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.502968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.502952 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.503244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.503223 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwsx\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.503380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.503366 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.503529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.503516 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.522248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.504547 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.522248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.504795 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.522248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.513541 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.522248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.513595 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:38:03.522248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.517863 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podStartSLOduration=2.517846074 podStartE2EDuration="2.517846074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.393701589 +0000 UTC m=+1237.367036257" watchObservedRunningTime="2026-11-02 06:38:03.517846074 +0000 UTC m=+1237.491180759" Nov 02 06:38:03.531012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.527866 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podStartSLOduration=2.527850622 podStartE2EDuration="2.527850622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.525543124 +0000 UTC m=+1237.498877819" watchObservedRunningTime="2026-11-02 06:38:03.527850622 +0000 UTC m=+1237.501185308" Nov 02 06:38:03.548440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.547530 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:38:03.548440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.547668 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.601411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.600267 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5eed1bfb-dockercfg-rpwf4\"" Nov 02 06:38:03.619602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.619023 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.619602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.619198 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwvr\" (UniqueName: \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.631277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.624438 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5eed1bfb\"" Nov 02 06:38:03.631277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.630584 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.631956 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.631928 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwsx\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.700106 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.692501 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:03.727895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.727855 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwvr\" (UniqueName: \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.728247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.728202 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.729221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.729192 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.809012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.808739 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwvr\" (UniqueName: \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\") pod \"payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:03.813976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.813946 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:03.860322 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.829129 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.860322 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.829196 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:03.959056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.958099 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.959056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:03.958173 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:04.019059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.018261 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:04.131498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.131463 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.263518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.263478 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:04.264204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.210448 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.264400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.264378 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:04.833011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.825510 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.833011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.825589 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:04.959014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.958473 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.959014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.958533 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:04.959014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:04.959011 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.105082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.103342 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.105082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.103403 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:05.114597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.114567 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:38:05.185642 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.185574 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.185906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.185840 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:05.196123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.190774 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:05.196697 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.196665 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:05.200009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.196875 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:05.308359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.308318 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.554148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.554102 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.708707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.637080 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.952013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.939132 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.952013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.654084 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 06:38:05.952013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.654121 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.952013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:05.637118 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:06.084499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.084437 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:38:06.147012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.135486 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:06.177120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.164225 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:06.177120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.165273 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:06.221103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.217055 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.308018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.299923 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:06.332033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.221909 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.332033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.322700 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:06.418014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.393207 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.418381 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.418342 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:06.477119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.456978 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.477119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.457102 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:06.575752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.575687 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.737396 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.731895 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.737396 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.732887 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:06.932190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.932085 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 06:38:06.940015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.937803 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:06.940015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.937848 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.964709 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.964761 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.967651 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.967698 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.968479 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.969238 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.969416 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:06.971594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:06.969573 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 06:38:07.110424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.094930 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.110424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.095007 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:07.215938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.201889 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.215938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.203058 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:07.219857 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:07.219826 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f2bb8_f27f_4383_978e_ad4e0e58b220.slice/crio-b459e505294155783ef4a43462abcab5b6c68ee26403ac6a11b4c691a13e51d5 WatchSource:0}: Error finding container b459e505294155783ef4a43462abcab5b6c68ee26403ac6a11b4c691a13e51d5: Status 404 returned error can't find the container with id b459e505294155783ef4a43462abcab5b6c68ee26403ac6a11b4c691a13e51d5 Nov 02 06:38:07.229353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.229113 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:07.243677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.243648 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.243873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.243847 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:07.371410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.349091 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.637143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.637030 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.698472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.698432 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" event={"ID":"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472","Type":"ContainerStarted","Data":"78453388b6bb4005fdb79a5aaf6d6c0e2794e8902d6d7fd9b4b57d5545a0d59c"} Nov 02 06:38:07.706157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.706122 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:38:07.760401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.699502 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:07.761249 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.735522 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:07.815514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.815474 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-14448910\"" Nov 02 06:38:07.841048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.817835 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-14448910-dockercfg-59cr5\"" Nov 02 06:38:07.841048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.818408 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.841048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.818455 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:07.866011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.865919 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:38:07.866177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866021 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:38:07.866177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866053 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:38:07.866177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866116 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:38:07.866177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866149 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866182 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866225 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866258 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866290 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866322 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:38:07.866498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.866355 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:38:07.983071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:07.970737 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:38:08.002758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.002706 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:08.003253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.003224 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp99\" (UniqueName: \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:08.009033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.008981 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.905324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.022032 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:08.938068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.022087 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 06:38:09.139709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.022108 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:09.148445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.023191 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 02 06:38:09.148674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.023222 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 06:38:09.149131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.154648 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.149327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.149289 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:09.149688 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.519814 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp99\" (UniqueName: \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:09.154505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.154477 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:09.160191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.223653 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.160386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.160347 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:09.190323 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.767751 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp99\" (UniqueName: \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:09.190551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.821848 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:09.191044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.978886 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.195240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.520368 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:09.195662 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.520422 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:09.247429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.247393 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\") pod \"payload-processing-e2e-derive-14448910-7cfb66697-lps8g\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:09.247658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.520441 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:09.252012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.716265 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:38:09.304016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.517035 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.304383 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.304318 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:09.304842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:08.824024 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.305022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.304976 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:09.363907 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.363866 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:09.399685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.397383 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.399685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.397440 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:09.399685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.398732 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.399685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.398770 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:09.572011 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:09.563708 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:38:09.572219 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.572182 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.572301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.572239 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:09.573012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.572717 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.766013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.727232 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwhx" podUID="b465618f-6a18-431a-956a-61f3fb5e34bd" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.823114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.822472 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.823114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.822526 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:09.860497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.859336 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.933012 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:09.907733 6273 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.683s" Nov 02 06:38:09.956046 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:09.950671 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:10.073770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.045684 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.073770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.045747 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:10.185012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.143147 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:10.712887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.148107 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.718017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.714381 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:11.161013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.143590 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.161013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.202084 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.205191 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.205234 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.205259 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.696319 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.705422 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.214250 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.171643 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.160182 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.913550 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.172020 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.006868 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" event={"ID":"b68f2bb8-f27f-4383-978e-ad4e0e58b220","Type":"ContainerStarted","Data":"b459e505294155783ef4a43462abcab5b6c68ee26403ac6a11b4c691a13e51d5"} Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.037250 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.791545 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.791793 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.911550 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.113331 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.172431 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.184133 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.213465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.184209 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:11.226434 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:10.187043 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.402248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.253167 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.402248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.402192 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:11.402493 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.292620 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.799063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:11.797099 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.176092 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.176497 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.176770 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.177067 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.177322 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:12.214080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.177552 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.263117 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.263500 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.264640 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.264679 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.264749 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:12.308013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.264770 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:12.392110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.369902 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.531531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.446106 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:38:12.800521 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.800414 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.917170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.847101 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:38:12.917170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.860179 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.962186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.961443 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:38:12.962186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:12.961507 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:13.211029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.197746 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:13.230348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.223377 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:38:13.230348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.224182 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:13.232117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.231418 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 06:38:13.232117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.232006 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:13.248344 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.248311 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 06:38:13.248452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.248361 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 06:38:13.267745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.267699 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:13.291404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.291059 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:13.356219 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:13.356183 6273 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.047s" Nov 02 06:38:13.356411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356289 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:13.356411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356360 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:13.356411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356386 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" event={"ID":"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472","Type":"ContainerStarted","Data":"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89"} Nov 02 06:38:13.356574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356425 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:38:13.356574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356494 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:38:13.356677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356617 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:38:13.356677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356672 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:38:13.356789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356707 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:38:13.356789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356740 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:13.356789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356773 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:38:13.356948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356865 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:38:13.356948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.356914 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:13.367950 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.365453 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:13.433169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.432663 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:38:13.438169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.436666 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:13.438169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.436721 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:38:13.438169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.436758 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:38:13.438169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.436821 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:38:13.438169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.437247 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:13.448626 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.448599 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:38:13.450150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.450120 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" event={"ID":"b68f2bb8-f27f-4383-978e-ad4e0e58b220","Type":"ContainerStarted","Data":"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0"} Nov 02 06:38:13.450248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.450175 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:13.456622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.456600 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:38:13.457497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.457479 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:38:13.696485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.696434 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" podStartSLOduration=11.696417480000001 podStartE2EDuration="11.69641748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:13.523674398 +0000 UTC m=+1247.497009083" watchObservedRunningTime="2026-11-02 06:38:13.69641748 +0000 UTC m=+1247.669752164" Nov 02 06:38:13.699724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.699684 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:13.700840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.700812 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 06:38:13.700942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:13.700867 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 06:38:14.224687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.224653 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:14.226589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.226465 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:14.228767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.228740 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:38:14.484650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.484558 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" event={"ID":"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f","Type":"ContainerStarted","Data":"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d"} Nov 02 06:38:14.484650 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.484620 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" event={"ID":"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f","Type":"ContainerStarted","Data":"f3c3f9ca58b60c3f82ea734926b8c6659bd2c6d04b35501b90380fc8088c5508"} Nov 02 06:38:14.485213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.484908 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:14.700211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.700180 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 06:38:14.700347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:14.700232 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 06:38:15.094640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:15.094572 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" podStartSLOduration=12.094558768 podStartE2EDuration="12.094558768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:15.092464492 +0000 UTC m=+1249.065799190" watchObservedRunningTime="2026-11-02 06:38:15.094558768 +0000 UTC m=+1249.067893453" Nov 02 06:38:15.435982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:15.435920 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" podStartSLOduration=10.435902069 podStartE2EDuration="10.435902069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:15.422698211 +0000 UTC m=+1249.396032895" watchObservedRunningTime="2026-11-02 06:38:15.435902069 +0000 UTC m=+1249.409236790" Nov 02 06:38:15.496546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:15.496511 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:38:15.700008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:15.699909 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 06:38:15.700008 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:15.699961 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 06:38:16.772424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:16.772388 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:16.772982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:16.772955 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:16.774795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:16.774774 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:38:16.952775 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:16.952745 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:38:16.957169 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:16.957152 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.013769 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.013735 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:38:17.026083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.025972 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026070 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026169 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026330 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026289 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026325 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026411 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026530 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026511 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026720 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.026790 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.026764 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128091 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128051 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128091 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128095 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128153 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128186 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128331 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128476 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128568 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128511 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128606 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128719 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.128875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.128854 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.129188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.129163 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.129474 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.129450 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.129558 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.129477 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.129558 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.129510 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.130926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.130903 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.131597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.131574 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.185361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.185335 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.187102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.187080 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.270509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.270486 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:17.703720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.703686 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:38:17.709145 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:17.709120 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bfb9b4_8fad_443c_9889_7c50080f6a50.slice/crio-e93280bf0345d9d1fae1bb7a4a7b6e2e4d52ff2d1ea82c874acdba7a53bab8d7 WatchSource:0}: Error finding container e93280bf0345d9d1fae1bb7a4a7b6e2e4d52ff2d1ea82c874acdba7a53bab8d7: Status 404 returned error can't find the container with id e93280bf0345d9d1fae1bb7a4a7b6e2e4d52ff2d1ea82c874acdba7a53bab8d7 Nov 02 06:38:17.711653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.711618 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:17.711806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.711700 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:17.711806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:17.711747 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:18.519473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:18.519432 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" event={"ID":"f0bfb9b4-8fad-443c-9889-7c50080f6a50","Type":"ContainerStarted","Data":"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d"} Nov 02 06:38:18.519892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:18.519497 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" event={"ID":"f0bfb9b4-8fad-443c-9889-7c50080f6a50","Type":"ContainerStarted","Data":"e93280bf0345d9d1fae1bb7a4a7b6e2e4d52ff2d1ea82c874acdba7a53bab8d7"} Nov 02 06:38:18.636009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:18.635940 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podStartSLOduration=2.635923637 podStartE2EDuration="2.635923637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:18.634400028 +0000 UTC m=+1252.607734713" watchObservedRunningTime="2026-11-02 06:38:18.635923637 +0000 UTC m=+1252.609258324" Nov 02 06:38:19.273011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:19.272674 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:19.273011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:19.272840 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:19.273281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:19.273009 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:20.273016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:20.272504 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:20.273016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:20.272574 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:21.051049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.044115 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.146013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.145076 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:21.148014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.146352 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 06:38:21.148014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.146718 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:21.156009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.152090 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.191011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.190075 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.191011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.190965 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:21.210024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.208062 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.210024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.208456 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:21.263011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.262646 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:21.284013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.274349 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:21.284013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.274399 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:21.418016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.407067 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.882116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:21.880974 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.151719 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.152208 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.152970 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.153707 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.153765 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:38:22.160071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.153803 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:38:22.218016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.217077 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.271013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.224716 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:22.271013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.226198 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.271013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.241746 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 06:38:22.285371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.285330 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:22.349162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.315879 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:22.349162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.315951 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:22.349162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.322131 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.424119 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495168 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495234 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495278 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495312 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495347 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495381 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:22.541301 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.495416 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:38:22.885195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:22.885082 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.078402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.078357 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:38:23.155117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.155068 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 06:38:23.251433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.155186 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:23.333014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.288872 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.380813 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.378882 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:23.385050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.155241 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 06:38:23.413175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.336590 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.413400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.310250 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:23.429019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.421214 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:23.429019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.421299 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:23.467015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.453119 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.483198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.479098 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 06:38:23.547241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.486571 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 06:38:23.643022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.629526 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 02 06:38:24.075151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:23.899898 6273 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.303229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.173352 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.303229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.202455 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.303229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.283336 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 02 06:38:24.374802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.373889 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.389805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.389587 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:24.398013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.396665 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:24.398013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.396743 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:24.530391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.530342 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.555106 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.555059 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 06:38:24.562777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.561248 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.562777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.561862 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 06:38:24.566221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.564053 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.638046 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.637360 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.713975 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.713942 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:38:24.714234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.714216 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 06:38:24.716795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.716773 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:38:24.800019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.798538 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:38:24.802648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.802628 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:38:24.808203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.808185 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:38:24.837056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.837032 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:38:24.865903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.865877 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 06:38:24.888543 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.888472 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:38:24.911024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.906833 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 06:38:24.911024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:24.908944 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 06:38:25.280487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:25.280449 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:25.280638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:25.280519 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:25.501765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:25.501722 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:38:26.271453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:26.271420 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:38:26.271622 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:26.271485 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 06:38:26.370186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:26.370152 6273 scope.go:122] "RemoveContainer" containerID="3d2d9d5ffca2016d4b976d66d3121ab0d035bccaf323691a773cbccf49fc6c41" Nov 02 06:38:26.384408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:26.384379 6273 scope.go:122] "RemoveContainer" containerID="2fdd848eeaf24c70c343fa3f5a313de3fb9fe4631cdf68c47db93a74496987f8" Nov 02 06:38:26.399604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:26.399326 6273 scope.go:122] "RemoveContainer" containerID="2fdec35bb705ff23e5c70578d8e0e0c75e9e1ea9393a098767a4e283bb75202c" Nov 02 06:38:27.672359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:27.672328 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:27.672971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:27.672773 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:27.673725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:27.673705 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:38:33.979688 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:33.979606 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:38:33.985100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:33.985075 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.037790 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.037763 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:38:34.073862 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.073826 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.074071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.073869 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.074071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.074042 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp64m\" (UniqueName: \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.074209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.074073 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.080184 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.080160 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15867003-serving-cert\"" Nov 02 06:38:34.175509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.175476 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.175509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.175515 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.175766 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:34.175607 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15867003-serving-cert: secret "maas-api-e2e-ait-15867003-serving-cert" not found Nov 02 06:38:34.175766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.175647 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp64m\" (UniqueName: \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.175766 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:34.175670 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls podName:cf8689b6-557c-4cd4-b7f4-58dde56d670c nodeName:}" failed. No retries permitted until 2026-11-02 06:38:34.675655004 +0000 UTC m=+1268.648989671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls") pod "maas-api-e2e-ait-15867003-6856f595c9-x976j" (UID: "cf8689b6-557c-4cd4-b7f4-58dde56d670c") : secret "maas-api-e2e-ait-15867003-serving-cert" not found Nov 02 06:38:34.175766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.175711 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.176501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.176479 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.178182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.178158 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.235175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.235057 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp64m\" (UniqueName: \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.684286 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.684247 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.686670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.686640 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") pod \"maas-api-e2e-ait-15867003-6856f595c9-x976j\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:34.898636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:34.898599 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:35.283616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:35.283589 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:38:35.286919 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:35.286881 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8689b6_557c_4cd4_b7f4_58dde56d670c.slice/crio-13e4c732ec0710b6e1d2f06bd21d11c1368278f0726e183ceffc446059723bc9 WatchSource:0}: Error finding container 13e4c732ec0710b6e1d2f06bd21d11c1368278f0726e183ceffc446059723bc9: Status 404 returned error can't find the container with id 13e4c732ec0710b6e1d2f06bd21d11c1368278f0726e183ceffc446059723bc9 Nov 02 06:38:35.288913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:35.288884 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:38:35.713070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:35.713021 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" event={"ID":"cf8689b6-557c-4cd4-b7f4-58dde56d670c","Type":"ContainerStarted","Data":"13e4c732ec0710b6e1d2f06bd21d11c1368278f0726e183ceffc446059723bc9"} Nov 02 06:38:36.327566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.327541 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85"] Nov 02 06:38:36.331955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.331932 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.386359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.386338 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:38:36.414549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414524 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414690 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414559 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414690 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414636 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3890b910-f647-455f-adb3-3d26ebbde75c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414690 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414673 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414745 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414781 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxjr\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-kube-api-access-mpxjr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.414964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.414944 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3890b910-f647-455f-adb3-3d26ebbde75c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.415098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.415043 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.415167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.415147 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.435232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.435200 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85"] Nov 02 06:38:36.516324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516243 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516344 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516377 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516422 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3890b910-f647-455f-adb3-3d26ebbde75c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516470 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516520 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516573 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxjr\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-kube-api-access-mpxjr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516607 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516648 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516752 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3890b910-f647-455f-adb3-3d26ebbde75c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516826 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.516916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.516844 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.517109 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.517089 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.517470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.517447 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3890b910-f647-455f-adb3-3d26ebbde75c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.518753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.518734 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3890b910-f647-455f-adb3-3d26ebbde75c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.518904 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.518884 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3890b910-f647-455f-adb3-3d26ebbde75c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.570204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.570177 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxjr\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-kube-api-access-mpxjr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.570423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.570400 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3890b910-f647-455f-adb3-3d26ebbde75c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qr85\" (UID: \"3890b910-f647-455f-adb3-3d26ebbde75c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.645902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.645873 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:36.737325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.737269 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" event={"ID":"cf8689b6-557c-4cd4-b7f4-58dde56d670c","Type":"ContainerStarted","Data":"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149"} Nov 02 06:38:36.737562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.737540 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:36.846378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:36.846280 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" podStartSLOduration=2.905648829 podStartE2EDuration="3.846266232s" totalImagesPullingTime="940.617403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:36.84425985 +0000 UTC m=+1270.817594533" watchObservedRunningTime="2026-11-02 06:38:36.846266232 +0000 UTC m=+1270.819600915" Nov 02 06:38:37.075829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.075794 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85"] Nov 02 06:38:37.080142 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:37.080115 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3890b910_f647_455f_adb3_3d26ebbde75c.slice/crio-161a414d75d6e8bb98816ab8604075420aab01e86d9cd88e0dcad41622488e07 WatchSource:0}: Error finding container 161a414d75d6e8bb98816ab8604075420aab01e86d9cd88e0dcad41622488e07: Status 404 returned error can't find the container with id 161a414d75d6e8bb98816ab8604075420aab01e86d9cd88e0dcad41622488e07 Nov 02 06:38:37.083929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.083886 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.084713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.083980 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.084713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.084043 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.781472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.781426 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" event={"ID":"3890b910-f647-455f-adb3-3d26ebbde75c","Type":"ContainerStarted","Data":"06a48f61b7998f2e4358601bf15cd41d9b8367f914208a373b80103fe7ad8531"} Nov 02 06:38:37.782028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.781531 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" event={"ID":"3890b910-f647-455f-adb3-3d26ebbde75c","Type":"ContainerStarted","Data":"161a414d75d6e8bb98816ab8604075420aab01e86d9cd88e0dcad41622488e07"} Nov 02 06:38:37.892048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:37.891961 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podStartSLOduration=1.891947505 podStartE2EDuration="1.891947505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:37.89130791 +0000 UTC m=+1271.864642615" watchObservedRunningTime="2026-11-02 06:38:37.891947505 +0000 UTC m=+1271.865282189" Nov 02 06:38:38.587164 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:38.587115 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:38:38.646686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:38.646657 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:38.671011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:38.648956 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:38.671011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:38.649042 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:39.647689 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:39.647316 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:39.647689 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:39.647379 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:40.646349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:40.646310 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:40.646561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:40.646373 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:41.660014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:41.646303 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:41.660014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:41.646367 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:42.646159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:42.646122 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:42.646353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:42.646183 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:42.793963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:42.793933 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:38:43.646566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:43.646528 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:43.646743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:43.646603 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:44.646752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:44.646222 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 06:38:44.646752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:44.646291 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" podUID="3890b910-f647-455f-adb3-3d26ebbde75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 06:38:45.856706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:45.856670 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:45.857178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:45.857088 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:45.857856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:45.857833 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qr85" Nov 02 06:38:47.185473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.185442 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:38:47.190203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.190185 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.321758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.321727 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrvx\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.321954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.321769 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.321954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.321885 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.321978 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.322094 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.322161 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322246 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.322232 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.322275 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.322404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.322353 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.379778 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.379745 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:38:47.423827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.423799 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrvx\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.423837 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424108 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424060 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424132 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424260 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424242 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424312 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424318 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424431 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424475 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424567 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424519 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424564 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424851 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.424962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.424933 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.425275 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.425245 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.427306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.427258 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.428202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.428176 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.477684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.477620 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.477838 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.477813 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrvx\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.506634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.506610 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:47.949302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.949268 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:38:47.954845 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:47.954815 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8051d6e_2e6c_4763_a660_bdc672eee185.slice/crio-dfd0d41673c7ea803b04bfebf2f83990ceec2c81c855fbccfa30685c22fb6fcd WatchSource:0}: Error finding container dfd0d41673c7ea803b04bfebf2f83990ceec2c81c855fbccfa30685c22fb6fcd: Status 404 returned error can't find the container with id dfd0d41673c7ea803b04bfebf2f83990ceec2c81c855fbccfa30685c22fb6fcd Nov 02 06:38:47.957688 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.957650 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:47.957767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.957721 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:47.957767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:47.957750 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:48.895595 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:48.895558 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" event={"ID":"a8051d6e-2e6c-4763-a660-bdc672eee185","Type":"ContainerStarted","Data":"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d"} Nov 02 06:38:48.896011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:48.895615 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" event={"ID":"a8051d6e-2e6c-4763-a660-bdc672eee185","Type":"ContainerStarted","Data":"dfd0d41673c7ea803b04bfebf2f83990ceec2c81c855fbccfa30685c22fb6fcd"} Nov 02 06:38:49.507630 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:49.507601 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:49.508814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:49.508779 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:49.508957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:49.508833 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:50.507605 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:50.507569 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:50.508028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:50.507634 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:51.507728 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:51.507692 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:51.508186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:51.507763 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:52.507285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.507250 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:52.507458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.507316 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:52.701804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.701751 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podStartSLOduration=5.7017354099999995 podStartE2EDuration="5.70173541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:49.073313901 +0000 UTC m=+1283.046648588" watchObservedRunningTime="2026-11-02 06:38:52.70173541 +0000 UTC m=+1286.675070096" Nov 02 06:38:52.704653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.704633 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:38:52.709377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.709357 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.735474 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.735450 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.735574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.735502 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.735652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.735631 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhcf\" (UniqueName: \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.735762 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.735749 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.807835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.807758 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5eed1bfb-serving-cert\"" Nov 02 06:38:52.811666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.811639 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:38:52.837673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.837637 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhcf\" (UniqueName: \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.837825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.837804 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.838063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.838041 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.838137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.838111 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.838499 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:52.838478 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-serving-cert: secret "maas-api-e2e-migrate-5eed1bfb-serving-cert" not found Nov 02 06:38:52.848098 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:38:52.840104 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls podName:05aa2a5e-5344-4aee-935e-7e7bd4e85e72 nodeName:}" failed. No retries permitted until 2026-11-02 06:38:53.340082239 +0000 UTC m=+1287.313416904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls") pod "maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" (UID: "05aa2a5e-5344-4aee-935e-7e7bd4e85e72") : secret "maas-api-e2e-migrate-5eed1bfb-serving-cert" not found Nov 02 06:38:52.848098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.838938 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.848098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.841075 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:52.891468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:52.891441 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhcf\" (UniqueName: \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:53.345220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:53.345189 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:53.347618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:53.347591 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") pod \"maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:53.507568 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:53.507533 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:53.507735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:53.507584 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:53.624093 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:53.623978 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:38:54.035112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:54.035078 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:38:54.040875 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:38:54.040847 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aa2a5e_5344_4aee_935e_7e7bd4e85e72.slice/crio-672cea6d060e12315928ffa33c4bb38ab95cf728a77f8ab7acd3aef7bcbe9554 WatchSource:0}: Error finding container 672cea6d060e12315928ffa33c4bb38ab95cf728a77f8ab7acd3aef7bcbe9554: Status 404 returned error can't find the container with id 672cea6d060e12315928ffa33c4bb38ab95cf728a77f8ab7acd3aef7bcbe9554 Nov 02 06:38:54.507969 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:54.507934 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:38:54.508129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:54.508000 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:38:54.950448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:54.949283 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" event={"ID":"05aa2a5e-5344-4aee-935e-7e7bd4e85e72","Type":"ContainerStarted","Data":"672cea6d060e12315928ffa33c4bb38ab95cf728a77f8ab7acd3aef7bcbe9554"} Nov 02 06:38:55.511291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:55.511263 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:55.511727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:55.511713 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:55.512350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:55.512332 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:38:55.968323 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:55.968287 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" event={"ID":"05aa2a5e-5344-4aee-935e-7e7bd4e85e72","Type":"ContainerStarted","Data":"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f"} Nov 02 06:38:56.077970 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:56.077914 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" podStartSLOduration=3.252819117 podStartE2EDuration="4.077901202s" totalImagesPullingTime="825.082085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:56.076031181 +0000 UTC m=+1290.049365868" watchObservedRunningTime="2026-11-02 06:38:56.077901202 +0000 UTC m=+1290.051235887" Nov 02 06:38:56.983510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:38:56.983476 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:39:00.479313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:00.479266 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:39:03.031289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:03.031241 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:39:03.202013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:03.177132 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:39:08.553531 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:39:08.553448 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:39:12.218471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.218438 6273 generic.go:356] "Generic (PLEG): container finished" podID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerID="afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084" exitCode=0 Nov 02 06:39:12.218943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.218479 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerDied","Data":"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084"} Nov 02 06:39:12.218943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.218860 6273 scope.go:122] "RemoveContainer" containerID="afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084" Nov 02 06:39:12.220340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.220306 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:12.220422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.220376 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:12.220422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:12.220409 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:13.227387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:13.227353 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerStarted","Data":"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d"} Nov 02 06:39:14.011358 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:14.011322 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:39:14.012784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:14.012760 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:14.012873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:14.012825 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 06:39:15.010982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:15.010948 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:15.011374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:15.011045 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 06:39:16.011722 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:16.011632 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:16.012390 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:16.012347 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 06:39:17.011244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:17.011208 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:17.011435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:17.011265 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 06:39:17.622052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:17.622009 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:39:18.011439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:18.011407 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:18.011601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:18.011465 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 06:39:19.015718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:19.015687 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:39:19.016658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:19.016635 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:39:21.652077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.652046 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm"] Nov 02 06:39:21.656665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.656648 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689437 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7nb\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-kube-api-access-2k7nb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689545 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689630 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689702 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689768 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689739 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689858 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.689943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689926 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.690016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.689962 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.690060 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.690039 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.756208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.756178 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm"] Nov 02 06:39:21.791245 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791210 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791265 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791409 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7nb\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-kube-api-access-2k7nb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791478 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791588 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791800 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791780 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.791956 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791927 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.792053 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.792119 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.792123 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791938 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.791869 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.792787 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.792822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.792796 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.794170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.794143 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.795035 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.795006 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.847014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.846976 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.847296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.847274 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7nb\" (UniqueName: \"kubernetes.io/projected/6b6c73eb-608b-4148-ba6d-0ccfff6306d6-kube-api-access-2k7nb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w98qm\" (UID: \"6b6c73eb-608b-4148-ba6d-0ccfff6306d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:21.970902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:21.970832 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:22.410582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:22.410555 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm"] Nov 02 06:39:22.448310 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:39:22.448283 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c73eb_608b_4148_ba6d_0ccfff6306d6.slice/crio-e80a47d8b0e1152796232314f395a71ba05144a222949f80e1f524c269614249 WatchSource:0}: Error finding container e80a47d8b0e1152796232314f395a71ba05144a222949f80e1f524c269614249: Status 404 returned error can't find the container with id e80a47d8b0e1152796232314f395a71ba05144a222949f80e1f524c269614249 Nov 02 06:39:22.451316 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:22.451283 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:22.451415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:22.451360 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:22.451477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:22.451419 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:23.325853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.325805 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" event={"ID":"6b6c73eb-608b-4148-ba6d-0ccfff6306d6","Type":"ContainerStarted","Data":"edbf736acb56e05c5efd114c89befd06aaaf3fb001e69cc4748e9741d82fdfc9"} Nov 02 06:39:23.326286 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.325906 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" event={"ID":"6b6c73eb-608b-4148-ba6d-0ccfff6306d6","Type":"ContainerStarted","Data":"e80a47d8b0e1152796232314f395a71ba05144a222949f80e1f524c269614249"} Nov 02 06:39:23.486183 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.486129 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podStartSLOduration=2.486114458 podStartE2EDuration="2.486114458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:23.483052564 +0000 UTC m=+1317.456387248" watchObservedRunningTime="2026-11-02 06:39:23.486114458 +0000 UTC m=+1317.459449140" Nov 02 06:39:23.973240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.973194 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:23.974641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.974613 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:23.974764 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:23.974676 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:24.974130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:24.974085 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:24.974627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:24.974148 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:25.972025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:25.971315 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:25.972025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:25.971406 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:26.439317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:26.439289 6273 scope.go:122] "RemoveContainer" containerID="003c9cd66d0c3302f6baebebff98454c3e15a5588aa6f24ff4d0b8dcabaa104b" Nov 02 06:39:26.474778 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:26.474757 6273 scope.go:122] "RemoveContainer" containerID="2d84f8e741c820f762f3b16af5601769e442232612e8315ee2e365b4c3bb4ae7" Nov 02 06:39:26.973427 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:26.973391 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:26.973575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:26.973451 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:27.973379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:27.973340 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:27.973844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:27.973404 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:28.971673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:28.971634 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:28.971851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:28.971695 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:29.972005 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:29.971963 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w98qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 06:39:29.972478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:29.972033 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" podUID="6b6c73eb-608b-4148-ba6d-0ccfff6306d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 06:39:30.975278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:30.975248 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:30.975750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:30.975578 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:30.976430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:30.976404 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w98qm" Nov 02 06:39:38.554857 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:39:38.554817 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:39:48.570398 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:48.570363 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:39:58.678585 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.678553 6273 generic.go:356] "Generic (PLEG): container finished" podID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerID="2c9fa6a804bea5a58eadaf389b8c662fa3f18263824a84cfa69e3382e72c05d7" exitCode=0 Nov 02 06:39:58.679138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.678627 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerDied","Data":"2c9fa6a804bea5a58eadaf389b8c662fa3f18263824a84cfa69e3382e72c05d7"} Nov 02 06:39:58.679289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.679233 6273 scope.go:122] "RemoveContainer" containerID="2c9fa6a804bea5a58eadaf389b8c662fa3f18263824a84cfa69e3382e72c05d7" Nov 02 06:39:58.681158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.681129 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:58.681237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.681207 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:58.681274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:58.681239 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:59.688810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:39:59.688766 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerStarted","Data":"69a04a8def09e8b94e0fad56298f9ae163589ee2de2fb0fbff72d43026b0d443"} Nov 02 06:40:00.397463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:00.397420 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:00.397654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:00.397571 6273 patch_prober.go:29] interesting pod/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:00.397654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:00.397618 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 06:40:01.397864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:01.397819 6273 patch_prober.go:29] interesting pod/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:01.398304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:01.397897 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 06:40:02.397717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:02.397677 6273 patch_prober.go:29] interesting pod/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:02.397895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:02.397739 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 06:40:03.398445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:03.398415 6273 patch_prober.go:29] interesting pod/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:03.398865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:03.398486 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 06:40:03.647604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:03.647568 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:40:04.398269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:04.398238 6273 patch_prober.go:29] interesting pod/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:04.398446 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:04.398289 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 06:40:05.403596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:05.403562 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:05.404056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:05.404020 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:05.404909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:05.404887 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:08.553508 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:08.553469 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:40:08.648599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.648534 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" podUID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" containerName="payload-processing" containerID="cri-o://da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d" gracePeriod=25 Nov 02 06:40:08.935034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.935008 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:40:08.987232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.987207 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\" (UniqueName: \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\") pod \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " Nov 02 06:40:08.987397 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.987312 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\") pod \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\" (UID: \"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f\") " Nov 02 06:40:08.987702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.987677 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume" pod "146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" (UID: "146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:08.987802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.987788 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:08.989322 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:08.989298 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99" pod "146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" (UID: "146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f"). InnerVolumeSpecName "kube-api-access-ntp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:09.090366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.089956 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntp99\" (UniqueName: \"kubernetes.io/projected/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f-kube-api-access-ntp99\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:09.224812 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.224763 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:40:09.765926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.765891 6273 generic.go:356] "Generic (PLEG): container finished" podID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" containerID="da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d" exitCode=0 Nov 02 06:40:09.766409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.765954 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" Nov 02 06:40:09.766409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.765972 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" event={"ID":"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f","Type":"ContainerDied","Data":"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d"} Nov 02 06:40:09.766409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.766099 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g" event={"ID":"146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f","Type":"ContainerDied","Data":"f3c3f9ca58b60c3f82ea734926b8c6659bd2c6d04b35501b90380fc8088c5508"} Nov 02 06:40:09.766409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.766127 6273 scope.go:122] "RemoveContainer" containerID="da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d" Nov 02 06:40:09.779708 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.779685 6273 scope.go:122] "RemoveContainer" containerID="da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d" Nov 02 06:40:09.780068 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:09.780038 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d\": container with ID starting with da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d not found: ID does not exist" containerID="da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d" Nov 02 06:40:09.780195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.780074 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d"} err="failed to get container status \"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d\": rpc error: code = NotFound desc = could not find container \"da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d\": container with ID starting with da17d4d47b1994c8e2af6e6758bdc65741227c7a6df14fc73437149cfb0ac15d not found: ID does not exist" Nov 02 06:40:09.922781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.922742 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:40:09.930121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:09.930096 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-14448910-7cfb66697-lps8g"] Nov 02 06:40:10.181153 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:10.181119 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" path="/var/lib/kubelet/pods/146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f/volumes" Nov 02 06:40:14.226141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.226103 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" podUID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" containerName="payload-processing" containerID="cri-o://d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0" gracePeriod=25 Nov 02 06:40:14.507576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.507551 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:40:14.585839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.585811 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\" (UniqueName: \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\") pod \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " Nov 02 06:40:14.585973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.585955 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\") pod \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\" (UID: \"b68f2bb8-f27f-4383-978e-ad4e0e58b220\") " Nov 02 06:40:14.586300 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.586273 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume" pod "b68f2bb8-f27f-4383-978e-ad4e0e58b220" (UID: "b68f2bb8-f27f-4383-978e-ad4e0e58b220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:14.586660 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.586639 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f2bb8-f27f-4383-978e-ad4e0e58b220-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:14.587896 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.587873 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr" pod "b68f2bb8-f27f-4383-978e-ad4e0e58b220" (UID: "b68f2bb8-f27f-4383-978e-ad4e0e58b220"). InnerVolumeSpecName "kube-api-access-qxwvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:14.688352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.688319 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwvr\" (UniqueName: \"kubernetes.io/projected/b68f2bb8-f27f-4383-978e-ad4e0e58b220-kube-api-access-qxwvr\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:14.807523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.807440 6273 generic.go:356] "Generic (PLEG): container finished" podID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" containerID="d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0" exitCode=0 Nov 02 06:40:14.807523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.807483 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" Nov 02 06:40:14.807746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.807527 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" event={"ID":"b68f2bb8-f27f-4383-978e-ad4e0e58b220","Type":"ContainerDied","Data":"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0"} Nov 02 06:40:14.807746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.807585 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5" event={"ID":"b68f2bb8-f27f-4383-978e-ad4e0e58b220","Type":"ContainerDied","Data":"b459e505294155783ef4a43462abcab5b6c68ee26403ac6a11b4c691a13e51d5"} Nov 02 06:40:14.807746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.807615 6273 scope.go:122] "RemoveContainer" containerID="d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0" Nov 02 06:40:14.822659 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.822640 6273 scope.go:122] "RemoveContainer" containerID="d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0" Nov 02 06:40:14.822926 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:14.822907 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0\": container with ID starting with d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0 not found: ID does not exist" containerID="d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0" Nov 02 06:40:14.822978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.822934 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0"} err="failed to get container status \"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0\": rpc error: code = NotFound desc = could not find container \"d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0\": container with ID starting with d85551c65084f93250417f137e6cd87e69d2d55cff5e7e955e39bce46ac926e0 not found: ID does not exist" Nov 02 06:40:14.962503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.962472 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:40:14.968870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:14.968828 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5eed1bfb-6f9c75ffb9-rptq5"] Nov 02 06:40:16.181099 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:16.181065 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" path="/var/lib/kubelet/pods/b68f2bb8-f27f-4383-978e-ad4e0e58b220/volumes" Nov 02 06:40:26.835209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:26.835127 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:40:26.835600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:26.835357 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" podUID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" containerName="maas-api" containerID="cri-o://d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149" gracePeriod=30 Nov 02 06:40:27.146725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.146695 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:40:27.198102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.198063 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\" (UniqueName: \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\") pod \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " Nov 02 06:40:27.198304 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.198288 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\") pod \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " Nov 02 06:40:27.198371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.198337 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\") pod \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " Nov 02 06:40:27.198428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.198385 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") pod \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\" (UID: \"cf8689b6-557c-4cd4-b7f4-58dde56d670c\") " Nov 02 06:40:27.198793 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.198758 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca" pod "cf8689b6-557c-4cd4-b7f4-58dde56d670c" (UID: "cf8689b6-557c-4cd4-b7f4-58dde56d670c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:27.200247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.200182 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8689b6-557c-4cd4-b7f4-58dde56d670c-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:27.201180 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.201142 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs" pod "cf8689b6-557c-4cd4-b7f4-58dde56d670c" (UID: "cf8689b6-557c-4cd4-b7f4-58dde56d670c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:27.202298 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.202257 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls" pod "cf8689b6-557c-4cd4-b7f4-58dde56d670c" (UID: "cf8689b6-557c-4cd4-b7f4-58dde56d670c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:27.203818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.203791 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m" pod "cf8689b6-557c-4cd4-b7f4-58dde56d670c" (UID: "cf8689b6-557c-4cd4-b7f4-58dde56d670c"). InnerVolumeSpecName "kube-api-access-kp64m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:27.302095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.302012 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:27.302095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.302052 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp64m\" (UniqueName: \"kubernetes.io/projected/cf8689b6-557c-4cd4-b7f4-58dde56d670c-kube-api-access-kp64m\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:27.302095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.302068 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8689b6-557c-4cd4-b7f4-58dde56d670c-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:27.938147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.938112 6273 generic.go:356] "Generic (PLEG): container finished" podID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" containerID="d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149" exitCode=0 Nov 02 06:40:27.938648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.938239 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" Nov 02 06:40:27.938648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.938164 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" event={"ID":"cf8689b6-557c-4cd4-b7f4-58dde56d670c","Type":"ContainerDied","Data":"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149"} Nov 02 06:40:27.938648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.938546 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j" event={"ID":"cf8689b6-557c-4cd4-b7f4-58dde56d670c","Type":"ContainerDied","Data":"13e4c732ec0710b6e1d2f06bd21d11c1368278f0726e183ceffc446059723bc9"} Nov 02 06:40:27.938648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.938618 6273 scope.go:122] "RemoveContainer" containerID="d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149" Nov 02 06:40:27.955463 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.955439 6273 scope.go:122] "RemoveContainer" containerID="d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149" Nov 02 06:40:27.955818 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:27.955789 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149\": container with ID starting with d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149 not found: ID does not exist" containerID="d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149" Nov 02 06:40:27.955912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:27.955833 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149"} err="failed to get container status \"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149\": rpc error: code = NotFound desc = could not find container \"d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149\": container with ID starting with d9fd2c8673e02ec246ae31c7eca0d10e3f8fd4e4dec6660ba91b46a56d6d8149 not found: ID does not exist" Nov 02 06:40:28.102374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:28.102339 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:40:28.111135 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:28.111099 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15867003-6856f595c9-x976j"] Nov 02 06:40:28.186149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:28.185842 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" path="/var/lib/kubelet/pods/cf8689b6-557c-4cd4-b7f4-58dde56d670c/volumes" Nov 02 06:40:30.449336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:30.449279 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:40:30.449873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:30.449537 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" containerID="cri-o://683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d" gracePeriod=30 Nov 02 06:40:31.630280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630244 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:40:31.630909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630893 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" containerName="maas-api" Nov 02 06:40:31.630953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630910 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" containerName="maas-api" Nov 02 06:40:31.630953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630946 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" containerName="payload-processing" Nov 02 06:40:31.631063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630951 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" containerName="payload-processing" Nov 02 06:40:31.631063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630970 6273 cpu_manager.go:414] "containerMap: removing container" podUID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" containerName="payload-processing" Nov 02 06:40:31.631063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.630975 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" containerName="payload-processing" Nov 02 06:40:31.631176 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.631096 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="146e15f4-496b-4e97-aa1e-3c2fb2a2aa8f" containerName="payload-processing" Nov 02 06:40:31.631176 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.631109 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8689b6-557c-4cd4-b7f4-58dde56d670c" containerName="maas-api" Nov 02 06:40:31.631176 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.631121 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68f2bb8-f27f-4383-978e-ad4e0e58b220" containerName="payload-processing" Nov 02 06:40:31.636096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.636076 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.692254 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.692228 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc9fdd1c-openshift-default-dockercfg-xkdsd\"" Nov 02 06:40:31.742195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.742156 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:40:31.768061 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768025 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768108 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v86\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768231 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768213 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768289 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768445 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768497 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768528 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768681 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.768760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.768716 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.875354 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.875304 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.875633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.875612 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.875857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.875841 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.875980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.875966 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.876227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.876208 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.876372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.876358 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v86\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.876512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.876497 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.876628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.876614 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.876855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.876840 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.898064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.891835 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.898978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.895533 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.899237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.896269 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.899345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.896510 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.899442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.896719 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.910005 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.909611 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.919728 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.919668 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.952129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.952096 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v86\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:31.971012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:31.962556 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\") pod \"e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:32.251160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:32.251131 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:32.727724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:32.727686 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:40:32.732648 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:32.732622 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda074febf_476f_4962_92f7_8bdc3ecac195.slice/crio-47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5 WatchSource:0}: Error finding container 47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5: Status 404 returned error can't find the container with id 47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5 Nov 02 06:40:32.735268 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:32.735235 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:32.735339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:32.735305 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:32.735339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:32.735335 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:33.117636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:33.117551 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerStarted","Data":"c743ca3c5aa0ae00fcb5fbb2dbdbcb96f7be7e709da515554edfd419db30bb5b"} Nov 02 06:40:33.117636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:33.117617 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerStarted","Data":"47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5"} Nov 02 06:40:33.251720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:33.251687 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:33.257701 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:33.257675 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:33.279815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:33.279754 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podStartSLOduration=2.279734487 podStartE2EDuration="2.279734487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:33.276467036 +0000 UTC m=+1387.249801722" watchObservedRunningTime="2026-11-02 06:40:33.279734487 +0000 UTC m=+1387.253069172" Nov 02 06:40:34.018943 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:34.018901 6273 patch_prober.go:29] interesting pod/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:40:34.019364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:34.018981 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:40:34.126406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:34.126371 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:34.127346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:34.127323 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:40:35.778063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.764937 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:40:35.778063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.765248 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" podUID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" containerName="maas-api" containerID="cri-o://6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f" gracePeriod=30 Nov 02 06:40:35.879864 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:35.879817 6273 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aa2a5e_5344_4aee_935e_7e7bd4e85e72.slice/crio-6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aa2a5e_5344_4aee_935e_7e7bd4e85e72.slice/crio-conmon-6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f.scope\": RecentStats: unable to find data in memory cache]" Nov 02 06:40:35.983484 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.983456 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:40:35.991933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.991902 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992127 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.991960 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992127 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992017 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992127 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992100 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992129 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992171 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992224 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:35.992309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992254 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992421 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992457 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\") pod \"69dec9a4-b77d-4614-a21a-db0216a0bf55\" (UID: \"69dec9a4-b77d-4614-a21a-db0216a0bf55\") " Nov 02 06:40:35.992624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992555 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:35.993205 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.992882 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:35.993205 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993167 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:35.993205 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993175 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:35.993418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993393 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.993495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993421 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69dec9a4-b77d-4614-a21a-db0216a0bf55-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.993495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993437 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.993495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993451 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.993495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.993465 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.994540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.994517 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:35.994694 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.994678 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:35.995552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.995513 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "kube-api-access-fgh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:35.996345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:35.996319 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo" pod "69dec9a4-b77d-4614-a21a-db0216a0bf55" (UID: "69dec9a4-b77d-4614-a21a-db0216a0bf55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:40:36.094606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.094493 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgh9x\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-kube-api-access-fgh9x\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.094606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.094535 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.094606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.094552 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.094606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.094564 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69dec9a4-b77d-4614-a21a-db0216a0bf55-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.118467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.118436 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:40:36.156085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.153213 6273 generic.go:356] "Generic (PLEG): container finished" podID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" containerID="6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f" exitCode=0 Nov 02 06:40:36.156085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.153484 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" event={"ID":"05aa2a5e-5344-4aee-935e-7e7bd4e85e72","Type":"ContainerDied","Data":"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f"} Nov 02 06:40:36.156085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.153536 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" event={"ID":"05aa2a5e-5344-4aee-935e-7e7bd4e85e72","Type":"ContainerDied","Data":"672cea6d060e12315928ffa33c4bb38ab95cf728a77f8ab7acd3aef7bcbe9554"} Nov 02 06:40:36.156085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.153565 6273 scope.go:122] "RemoveContainer" containerID="6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f" Nov 02 06:40:36.156085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.153726 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5" Nov 02 06:40:36.239054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.238982 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") pod \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " Nov 02 06:40:36.239226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.239135 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\" (UniqueName: \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\") pod \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " Nov 02 06:40:36.239226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.239185 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\") pod \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " Nov 02 06:40:36.239368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.239270 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\") pod \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\" (UID: \"05aa2a5e-5344-4aee-935e-7e7bd4e85e72\") " Nov 02 06:40:36.249552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.249511 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf" pod "05aa2a5e-5344-4aee-935e-7e7bd4e85e72" (UID: "05aa2a5e-5344-4aee-935e-7e7bd4e85e72"). InnerVolumeSpecName "kube-api-access-vvhcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:36.269704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.269462 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls" pod "05aa2a5e-5344-4aee-935e-7e7bd4e85e72" (UID: "05aa2a5e-5344-4aee-935e-7e7bd4e85e72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:36.271063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.270060 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca" pod "05aa2a5e-5344-4aee-935e-7e7bd4e85e72" (UID: "05aa2a5e-5344-4aee-935e-7e7bd4e85e72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:36.283314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.283272 6273 generic.go:356] "Generic (PLEG): container finished" podID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerID="683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d" exitCode=0 Nov 02 06:40:36.284288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.284267 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" Nov 02 06:40:36.291407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.291355 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs" pod "05aa2a5e-5344-4aee-935e-7e7bd4e85e72" (UID: "05aa2a5e-5344-4aee-935e-7e7bd4e85e72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.303638 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerDied","Data":"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d"} Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.303720 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb" event={"ID":"69dec9a4-b77d-4614-a21a-db0216a0bf55","Type":"ContainerDied","Data":"e82bddcc7ddcb595621fea875bf2d9a632f3d1c8c1f81507ee3c699b4e2c0f8e"} Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.314601 6273 scope.go:122] "RemoveContainer" containerID="6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f" Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:36.315137 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f\": container with ID starting with 6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f not found: ID does not exist" containerID="6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f" Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.315178 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f"} err="failed to get container status \"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f\": rpc error: code = NotFound desc = could not find container \"6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f\": container with ID starting with 6d188b0a4c966a8b5ef114eaf21e8d309116aba27481ae86a0a7c1527a1fad5f not found: ID does not exist" Nov 02 06:40:36.317574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.315201 6273 scope.go:122] "RemoveContainer" containerID="683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d" Nov 02 06:40:36.336905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.336876 6273 scope.go:122] "RemoveContainer" containerID="afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084" Nov 02 06:40:36.353143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.353120 6273 scope.go:122] "RemoveContainer" containerID="683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d" Nov 02 06:40:36.353812 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:36.353629 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d\": container with ID starting with 683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d not found: ID does not exist" containerID="683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d" Nov 02 06:40:36.353812 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.353667 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d"} err="failed to get container status \"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d\": rpc error: code = NotFound desc = could not find container \"683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d\": container with ID starting with 683bf759a0bb34c185604da8091575881621b492d9e2c639e6ec9c392d488f4d not found: ID does not exist" Nov 02 06:40:36.353812 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.353737 6273 scope.go:122] "RemoveContainer" containerID="afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084" Nov 02 06:40:36.354400 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:36.354321 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084\": container with ID starting with afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084 not found: ID does not exist" containerID="afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084" Nov 02 06:40:36.354400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.354356 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084"} err="failed to get container status \"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084\": rpc error: code = NotFound desc = could not find container \"afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084\": container with ID starting with afe9c6cfcf2c8256c062e6b0145f36825830571c3d7fdc6668eb341dd00c4084 not found: ID does not exist" Nov 02 06:40:36.355043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.354982 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.355043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.355037 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhcf\" (UniqueName: \"kubernetes.io/projected/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-kube-api-access-vvhcf\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.355206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.355055 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.355206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.355069 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa2a5e-5344-4aee-935e-7e7bd4e85e72-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.438278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.438238 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:40:36.447939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.447886 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15867003-openshift-default-6c8f44b75c-6tmvb"] Nov 02 06:40:36.581089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.581056 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:40:36.585590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:36.585559 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5eed1bfb-5ddfdc7dbb-jjtw5"] Nov 02 06:40:37.213925 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.213884 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r"] Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215141 6273 cpu_manager.go:414] "containerMap: removing container" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215166 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215196 6273 cpu_manager.go:414] "containerMap: removing container" podUID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" containerName="maas-api" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215204 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" containerName="maas-api" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215220 6273 cpu_manager.go:414] "containerMap: removing container" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215228 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215431 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215448 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" containerName="maas-api" Nov 02 06:40:37.216686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.215457 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" containerName="istio-proxy" Nov 02 06:40:37.234014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.221774 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.276357 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r"] Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277447 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277533 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277602 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277715 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277762 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfklc\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-kube-api-access-cfklc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277824 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.277921 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.278172 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.286012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.278213 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.379839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.379796 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.379852 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfklc\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-kube-api-access-cfklc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.379915 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380025 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380146 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380178 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380245 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380301 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380305 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380388 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380398 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380557 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.380660 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.380640 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.381174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.381147 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.382903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.382873 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.383111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.383078 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.434159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.434128 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfklc\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-kube-api-access-cfklc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.434567 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.434547 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eef205e-a4bf-402c-83ce-f3bd42c0cf7c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6mt9r\" (UID: \"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.541308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.541226 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:37.981633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:37.981591 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r"] Nov 02 06:40:38.025330 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:38.025300 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eef205e_a4bf_402c_83ce_f3bd42c0cf7c.slice/crio-e7e061d2c411a5c642c1af8cf5719151ccc0fdca6fd64389b25bc15bac259a1e WatchSource:0}: Error finding container e7e061d2c411a5c642c1af8cf5719151ccc0fdca6fd64389b25bc15bac259a1e: Status 404 returned error can't find the container with id e7e061d2c411a5c642c1af8cf5719151ccc0fdca6fd64389b25bc15bac259a1e Nov 02 06:40:38.028372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.028329 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.028462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.028422 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.028498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.028469 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.183062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.183025 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05aa2a5e-5344-4aee-935e-7e7bd4e85e72" path="/var/lib/kubelet/pods/05aa2a5e-5344-4aee-935e-7e7bd4e85e72/volumes" Nov 02 06:40:38.183705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.183689 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69dec9a4-b77d-4614-a21a-db0216a0bf55" path="/var/lib/kubelet/pods/69dec9a4-b77d-4614-a21a-db0216a0bf55/volumes" Nov 02 06:40:38.305466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.305373 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" event={"ID":"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c","Type":"ContainerStarted","Data":"59c083e5c5144f2077b20840531377e632a5fdb64d10643723bc4da199ec952a"} Nov 02 06:40:38.305466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.305436 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" event={"ID":"7eef205e-a4bf-402c-83ce-f3bd42c0cf7c","Type":"ContainerStarted","Data":"e7e061d2c411a5c642c1af8cf5719151ccc0fdca6fd64389b25bc15bac259a1e"} Nov 02 06:40:38.419420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.419366 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podStartSLOduration=1.419350949 podStartE2EDuration="1.419350949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:38.418689763 +0000 UTC m=+1392.392024449" watchObservedRunningTime="2026-11-02 06:40:38.419350949 +0000 UTC m=+1392.392685641" Nov 02 06:40:38.541762 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:38.541720 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:38.566217 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:40:38.566121 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:40:39.224955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.224912 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:40:39.225603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.225567 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" containerID="cri-o://69a04a8def09e8b94e0fad56298f9ae163589ee2de2fb0fbff72d43026b0d443" gracePeriod=30 Nov 02 06:40:39.229563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.229536 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:40:39.240038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.236402 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.285851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.285816 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a25b83\"" Nov 02 06:40:39.285851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.285824 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a25b83-dockercfg-8mf6w\"" Nov 02 06:40:39.286491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.286470 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:40:39.312941 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.312903 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzn4\" (UniqueName: \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.313458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.313071 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.415148 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.415105 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.415466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.415447 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzn4\" (UniqueName: \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.416104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.416066 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.632203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.586604 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:39.632203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.586675 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:39.632203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.603262 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzn4\" (UniqueName: \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\") pod \"payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:39.875733 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:39.870243 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:40.279212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:40.279187 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:40:40.281871 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:40.281840 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02b12f5_2751_4469_8a4b_7b9c07d41776.slice/crio-c97b1e865337ac2afc81795420c93626438bfbb2477cfc08a44e56892c2a8e74 WatchSource:0}: Error finding container c97b1e865337ac2afc81795420c93626438bfbb2477cfc08a44e56892c2a8e74: Status 404 returned error can't find the container with id c97b1e865337ac2afc81795420c93626438bfbb2477cfc08a44e56892c2a8e74 Nov 02 06:40:40.326147 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:40.326118 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" event={"ID":"e02b12f5-2751-4469-8a4b-7b9c07d41776","Type":"ContainerStarted","Data":"c97b1e865337ac2afc81795420c93626438bfbb2477cfc08a44e56892c2a8e74"} Nov 02 06:40:40.542503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:40.542429 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:40.542503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:40.542478 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:41.335602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.335564 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" event={"ID":"e02b12f5-2751-4469-8a4b-7b9c07d41776","Type":"ContainerStarted","Data":"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295"} Nov 02 06:40:41.336036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.335632 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:41.444265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.444214 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" podStartSLOduration=2.444201027 podStartE2EDuration="2.444201027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:41.443400017 +0000 UTC m=+1395.416734702" watchObservedRunningTime="2026-11-02 06:40:41.444201027 +0000 UTC m=+1395.417535711" Nov 02 06:40:41.541580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.541550 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:41.541721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.541599 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:41.638739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.638664 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:40:41.643064 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.643043 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.733202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.729193 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3bf02ebf-openshift-default-dockercfg-njbqs\"" Nov 02 06:40:41.759011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.748282 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821698 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821751 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h969f\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821784 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821816 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821867 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821913 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.821970 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.822010 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.825013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.822073 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924335 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924380 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h969f\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924409 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924442 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924499 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924548 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924616 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924641 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924768 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.924947 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.924850 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.925264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.925242 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.925528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.925503 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.925773 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.925744 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.926322 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.926296 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.928259 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.928238 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:41.928451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:41.928431 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:42.005146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.005112 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:42.009470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.009406 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h969f\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:42.139514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.139478 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:42.542543 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.542455 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:42.542543 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.542508 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:42.616014 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:42.615969 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db22104_c2e8_4240_aee6_9074f29d3418.slice/crio-21e04247ddd93f783f10de6651a0ade08c5d6b0709bb3813b314221ec56ead6e WatchSource:0}: Error finding container 21e04247ddd93f783f10de6651a0ade08c5d6b0709bb3813b314221ec56ead6e: Status 404 returned error can't find the container with id 21e04247ddd93f783f10de6651a0ade08c5d6b0709bb3813b314221ec56ead6e Nov 02 06:40:42.618695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.618660 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:42.618762 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.618728 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:42.618812 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.618760 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:42.647309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:42.647279 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:40:43.379692 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:43.379644 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerStarted","Data":"944eaeaf2876d4d3e13a668b3db7e7487e96236d951981d91bf9b65f9d170802"} Nov 02 06:40:43.379889 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:43.379762 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerStarted","Data":"21e04247ddd93f783f10de6651a0ade08c5d6b0709bb3813b314221ec56ead6e"} Nov 02 06:40:43.542128 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:43.542093 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:43.542321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:43.542160 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:43.587854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:43.587797 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podStartSLOduration=2.587781049 podStartE2EDuration="2.587781049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:43.536479355 +0000 UTC m=+1397.509814038" watchObservedRunningTime="2026-11-02 06:40:43.587781049 +0000 UTC m=+1397.561115733" Nov 02 06:40:44.146071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.139804 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:44.147535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.147500 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:44.406071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.402220 6273 generic.go:356] "Generic (PLEG): container finished" podID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerID="69a04a8def09e8b94e0fad56298f9ae163589ee2de2fb0fbff72d43026b0d443" exitCode=0 Nov 02 06:40:44.406071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.405028 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerDied","Data":"69a04a8def09e8b94e0fad56298f9ae163589ee2de2fb0fbff72d43026b0d443"} Nov 02 06:40:44.406071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.405106 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:44.406071 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.405138 6273 scope.go:122] "RemoveContainer" containerID="2c9fa6a804bea5a58eadaf389b8c662fa3f18263824a84cfa69e3382e72c05d7" Nov 02 06:40:44.408590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.407003 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:40:44.559013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.541683 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:44.559013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.541738 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:44.745932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.745901 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:44.838370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838335 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838459 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838492 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838520 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838608 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838639 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:44.838723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838662 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838731 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.838875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838733 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:44.838954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838908 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:44.838954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.838919 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.839082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839024 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:44.839082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839059 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\") pod \"7f775a69-1941-45d4-ae5e-50638498c8e9\" (UID: \"7f775a69-1941-45d4-ae5e-50638498c8e9\") " Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839364 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839795 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f775a69-1941-45d4-ae5e-50638498c8e9-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839819 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839835 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839850 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.839891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.839865 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.840903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.840872 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:44.841201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.841181 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:40:44.841491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.841473 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:44.842104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.842086 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9" pod "7f775a69-1941-45d4-ae5e-50638498c8e9" (UID: "7f775a69-1941-45d4-ae5e-50638498c8e9"). InnerVolumeSpecName "kube-api-access-vq8w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:44.941637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.941559 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.941637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.941585 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.941637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.941596 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8w9\" (UniqueName: \"kubernetes.io/projected/7f775a69-1941-45d4-ae5e-50638498c8e9-kube-api-access-vq8w9\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:44.941637 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:44.941606 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f775a69-1941-45d4-ae5e-50638498c8e9-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:45.414048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.414019 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" Nov 02 06:40:45.414210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.414049 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t" event={"ID":"7f775a69-1941-45d4-ae5e-50638498c8e9","Type":"ContainerDied","Data":"631c93c81c10840fa182f94f878a416eaa817edc32cdcd128925698a6589ea98"} Nov 02 06:40:45.414210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.414116 6273 scope.go:122] "RemoveContainer" containerID="69a04a8def09e8b94e0fad56298f9ae163589ee2de2fb0fbff72d43026b0d443" Nov 02 06:40:45.542410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.542317 6273 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 06:40:45.542410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.542381 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" podUID="7eef205e-a4bf-402c-83ce-f3bd42c0cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 06:40:45.617592 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.617562 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:40:45.666101 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:45.666069 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5eed1bfb-gw-openshift-default-66d5966945-d456t"] Nov 02 06:40:46.183298 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:46.183265 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" path="/var/lib/kubelet/pods/7f775a69-1941-45d4-ae5e-50638498c8e9/volumes" Nov 02 06:40:46.881586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:46.881548 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:46.882016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:46.881981 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:46.882901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:46.882878 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6mt9r" Nov 02 06:40:49.922982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.922948 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:40:49.923769 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.923748 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:49.923816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.923771 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:49.923929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.923920 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:49.923964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.923934 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:49.933040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.929265 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:49.993549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:49.993516 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:40:50.031980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.031932 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3bf02ebf-dockercfg-wcdgb\"" Nov 02 06:40:50.031980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.031948 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3bf02ebf\"" Nov 02 06:40:50.036848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.036819 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4p4w\" (UniqueName: \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.040124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.038589 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.148900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.146688 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.148900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.146859 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4p4w\" (UniqueName: \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.148900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.148156 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.203972 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.203874 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4p4w\" (UniqueName: \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.250983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.250942 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:50.541092 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.538087 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:40:50.666738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:50.666703 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:40:50.672052 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:50.672020 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1606d8c1_f079_4b00_96a4_dcf91c28e2f8.slice/crio-14d2b64abf76741261842304098d628e31a7763c2ae6cfc1c179b3dc7bb31e19 WatchSource:0}: Error finding container 14d2b64abf76741261842304098d628e31a7763c2ae6cfc1c179b3dc7bb31e19: Status 404 returned error can't find the container with id 14d2b64abf76741261842304098d628e31a7763c2ae6cfc1c179b3dc7bb31e19 Nov 02 06:40:51.472269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:51.472225 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" event={"ID":"1606d8c1-f079-4b00-96a4-dcf91c28e2f8","Type":"ContainerStarted","Data":"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3"} Nov 02 06:40:51.472882 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:51.472853 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:40:51.473000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:51.472894 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" event={"ID":"1606d8c1-f079-4b00-96a4-dcf91c28e2f8","Type":"ContainerStarted","Data":"14d2b64abf76741261842304098d628e31a7763c2ae6cfc1c179b3dc7bb31e19"} Nov 02 06:40:51.584928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:51.584870 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" podStartSLOduration=2.584855246 podStartE2EDuration="2.584855246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:51.581395791 +0000 UTC m=+1405.554730499" watchObservedRunningTime="2026-11-02 06:40:51.584855246 +0000 UTC m=+1405.558189930" Nov 02 06:40:52.349816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:52.349784 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:40:56.128509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.128474 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:40:56.129269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.129246 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:56.129269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.129266 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f775a69-1941-45d4-ae5e-50638498c8e9" containerName="istio-proxy" Nov 02 06:40:56.133413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.133396 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.182561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.182537 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 06:40:56.182702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.182642 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 06:40:56.182702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.182649 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 06:40:56.182794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.182755 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk64l\"" Nov 02 06:40:56.233214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.233189 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:40:56.253607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253579 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.253746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253618 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.253746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253709 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.253746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253737 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.253897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253808 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qsr\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.253897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.253885 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.254098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.254082 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355360 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355491 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355536 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355595 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355620 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355665 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qsr\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.355783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.355740 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.356424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.356397 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.357930 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.357899 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.358079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.358059 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.358198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.358175 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.358237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.358214 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.419560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.419533 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qsr\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.419693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.419592 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4dqll\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.446707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.446686 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:40:56.930852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:56.930822 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:40:56.935211 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:40:56.935184 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd78ba7d_c1f4_4cc5_813c_007f6d13a96f.slice/crio-79ed72b16921d22b053ceb8f1c7ec7d5e74a329c83a6d6e75dac3d00ba5cc60e WatchSource:0}: Error finding container 79ed72b16921d22b053ceb8f1c7ec7d5e74a329c83a6d6e75dac3d00ba5cc60e: Status 404 returned error can't find the container with id 79ed72b16921d22b053ceb8f1c7ec7d5e74a329c83a6d6e75dac3d00ba5cc60e Nov 02 06:40:57.649345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:40:57.649310 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" event={"ID":"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f","Type":"ContainerStarted","Data":"79ed72b16921d22b053ceb8f1c7ec7d5e74a329c83a6d6e75dac3d00ba5cc60e"} Nov 02 06:41:01.640286 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:01.640248 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:01.640610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:01.640314 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:02.555042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.555013 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:41:02.698373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.697965 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" event={"ID":"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f","Type":"ContainerStarted","Data":"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329"} Nov 02 06:41:02.701523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.701401 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:41:02.708026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.707944 6273 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4dqll container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:41:02.708192 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.708060 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:41:02.812510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:02.812381 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" podStartSLOduration=2.110541099 podStartE2EDuration="6.812364039s" totalImagesPullingTime="4.70182294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:02.810129106 +0000 UTC m=+1416.783463804" watchObservedRunningTime="2026-11-02 06:41:02.812364039 +0000 UTC m=+1416.785698722" Nov 02 06:41:03.706447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:03.706391 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:41:08.554880 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:41:08.554845 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:41:37.013440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:37.013359 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:41:38.737798 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:41:38.737721 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:41:41.084121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:41.084075 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g"] Nov 02 06:41:41.103693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:41.103644 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e9465b-2pm8g"] Nov 02 06:41:42.185875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:42.185839 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a762ea5-ef86-4159-9e65-3c8ec57e8143" path="/var/lib/kubelet/pods/4a762ea5-ef86-4159-9e65-3c8ec57e8143/volumes" Nov 02 06:41:45.052243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:45.052207 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld"] Nov 02 06:41:45.098158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:45.098124 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbdb6ee-gq7ld"] Nov 02 06:41:46.183018 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:46.182957 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adf0f40-d9a6-4e2f-8c77-8c50263b1a62" path="/var/lib/kubelet/pods/6adf0f40-d9a6-4e2f-8c77-8c50263b1a62/volumes" Nov 02 06:41:48.309398 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.309296 6273 generic.go:356] "Generic (PLEG): container finished" podID="a074febf-476f-4962-92f7-8bdc3ecac195" containerID="c743ca3c5aa0ae00fcb5fbb2dbdbcb96f7be7e709da515554edfd419db30bb5b" exitCode=0 Nov 02 06:41:48.309853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.309399 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerDied","Data":"c743ca3c5aa0ae00fcb5fbb2dbdbcb96f7be7e709da515554edfd419db30bb5b"} Nov 02 06:41:48.309927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.309854 6273 scope.go:122] "RemoveContainer" containerID="c743ca3c5aa0ae00fcb5fbb2dbdbcb96f7be7e709da515554edfd419db30bb5b" Nov 02 06:41:48.311737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.311700 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:48.311963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.311795 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:48.311963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:48.311848 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:49.126618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:49.126570 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:41:49.320051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:49.320012 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerStarted","Data":"846e36c0671e50fe2087c2017fbeb4eb0ce4fdee8ad6239cbfd05dcd40d7368b"} Nov 02 06:41:50.266014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:50.260304 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:41:50.274123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:50.274091 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:41:50.274278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:50.274159 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 06:41:51.264012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:51.253340 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:41:51.264012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:51.253401 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 06:41:52.262402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:52.262362 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:41:52.262680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:52.262652 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 06:41:53.268011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:53.267413 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:41:53.268011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:53.267482 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 06:41:54.273134 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:54.273084 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:41:54.273648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:54.273172 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 06:41:55.256481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:55.256450 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:41:55.256845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:55.256822 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:41:55.257565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:55.257547 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:41:56.045897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:56.045861 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr"] Nov 02 06:41:56.092248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:56.092217 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d2eef6-pvjkr"] Nov 02 06:41:56.181389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:41:56.181359 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51213ce-c8a3-4956-ab31-f5ba9de58395" path="/var/lib/kubelet/pods/e51213ce-c8a3-4956-ab31-f5ba9de58395/volumes" Nov 02 06:42:04.052801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:04.052759 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb"] Nov 02 06:42:04.098827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:04.098796 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c4624a-kfrgb"] Nov 02 06:42:04.180777 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:04.180732 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31131fd-d4fb-4e6e-b85a-17936dae1d7a" path="/var/lib/kubelet/pods/e31131fd-d4fb-4e6e-b85a-17936dae1d7a/volumes" Nov 02 06:42:08.554096 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:42:08.554056 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:42:09.815184 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:09.815142 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:42:10.047047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:10.047004 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd"] Nov 02 06:42:10.050342 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:10.050315 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eacec68-9dkrd"] Nov 02 06:42:10.181957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:10.181924 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b27640-3773-456b-a5e6-e2c522b3302e" path="/var/lib/kubelet/pods/a3b27640-3773-456b-a5e6-e2c522b3302e/volumes" Nov 02 06:42:15.046782 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:15.046744 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b"] Nov 02 06:42:15.093405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:15.093348 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3c6760-6754b"] Nov 02 06:42:16.183900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:16.183866 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db615569-3a3a-4d81-aeaa-7f27b0d3043a" path="/var/lib/kubelet/pods/db615569-3a3a-4d81-aeaa-7f27b0d3043a/volumes" Nov 02 06:42:21.723901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.723869 6273 generic.go:356] "Generic (PLEG): container finished" podID="5db22104-c2e8-4240-aee6-9074f29d3418" containerID="944eaeaf2876d4d3e13a668b3db7e7487e96236d951981d91bf9b65f9d170802" exitCode=0 Nov 02 06:42:21.724382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.723938 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerDied","Data":"944eaeaf2876d4d3e13a668b3db7e7487e96236d951981d91bf9b65f9d170802"} Nov 02 06:42:21.724428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.724382 6273 scope.go:122] "RemoveContainer" containerID="944eaeaf2876d4d3e13a668b3db7e7487e96236d951981d91bf9b65f9d170802" Nov 02 06:42:21.726284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.726250 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:42:21.726353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.726339 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:42:21.726399 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:21.726383 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:42:22.733339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:22.733304 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerStarted","Data":"240694bc6ec8b716b36344fc466e25d17638e84648c2b910d4af1171aa42dff3"} Nov 02 06:42:23.139893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.139708 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:42:23.139893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.139879 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:42:23.140171 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.139930 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 06:42:23.541036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.541002 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2"] Nov 02 06:42:23.545254 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.545232 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.602303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.602261 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:42:23.602429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.602333 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:42:23.603905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.603879 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2"] Nov 02 06:42:23.702977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.702947 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wzr\" (UniqueName: \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.702977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.702997 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.703255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.703229 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.804888 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.804800 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.805363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.805005 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wzr\" (UniqueName: \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.805363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.805031 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.805468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.805446 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.807335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.807310 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.877476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.877441 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wzr\" (UniqueName: \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\") pod \"maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:23.887211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:23.887184 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:24.140524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:24.140429 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:42:24.140524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:24.140496 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 06:42:24.293700 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:24.293662 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2"] Nov 02 06:42:24.300515 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:42:24.300483 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c13f160_1d4c_4337_a4b4_5d75a0c30eb0.slice/crio-90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892 WatchSource:0}: Error finding container 90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892: Status 404 returned error can't find the container with id 90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892 Nov 02 06:42:24.776382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:24.776348 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" event={"ID":"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0","Type":"ContainerStarted","Data":"90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892"} Nov 02 06:42:25.140802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:25.140713 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:42:25.141209 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:25.140779 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 06:42:25.785880 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:25.785848 6273 generic.go:356] "Generic (PLEG): container finished" podID="1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" containerID="5da0c49ab5a4e6e12d6f8a65570671a94805fb1e2051d0480c2b67ddbe703df3" exitCode=0 Nov 02 06:42:25.786073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:25.785937 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" event={"ID":"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0","Type":"ContainerDied","Data":"5da0c49ab5a4e6e12d6f8a65570671a94805fb1e2051d0480c2b67ddbe703df3"} Nov 02 06:42:26.140529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.140492 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:42:26.140699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.140544 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 06:42:26.793684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.793193 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:42:26.799962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.799925 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.834074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.832941 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjvt\" (UniqueName: \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.834074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.833277 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.869011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.868559 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:42:26.897013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.896127 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-29abe7\"" Nov 02 06:42:26.897013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.896297 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-29abe7-dockercfg-lmnwb\"" Nov 02 06:42:26.969923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.960497 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjvt\" (UniqueName: \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.969923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.960769 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.969923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.961909 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:26.994576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:26.994534 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:42:27.008636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.004275 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.027602 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.027549 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:42:27.091424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.090214 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smv8q\" (UniqueName: \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.091424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.090345 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.144459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.142173 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:42:27.144459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.142235 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 06:42:27.163027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.152738 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjvt\" (UniqueName: \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\") pod \"payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:27.206022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.199111 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smv8q\" (UniqueName: \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.206022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.199228 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.206022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.200518 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.335376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.329183 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smv8q\" (UniqueName: \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\") pod \"payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.413608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.413573 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:27.415731 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.415700 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:27.526874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.526829 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\") pod \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " Nov 02 06:42:27.527010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.526970 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\") pod \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " Nov 02 06:42:27.528026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.527978 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca" pod "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" (UID: "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:42:27.528251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.528232 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\" (UniqueName: \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\") pod \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\" (UID: \"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0\") " Nov 02 06:42:27.528901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.528838 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:27.530193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.530159 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca" pod "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" (UID: "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:42:27.531521 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.531496 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr" pod "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" (UID: "1c13f160-1d4c-4337-a4b4-5d75a0c30eb0"). InnerVolumeSpecName "kube-api-access-s4wzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:42:27.542453 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.542157 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:27.631115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.630785 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4wzr\" (UniqueName: \"kubernetes.io/projected/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-kube-api-access-s4wzr\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:27.631115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.630821 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:27.811545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.811511 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" Nov 02 06:42:27.811545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.811536 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2" event={"ID":"1c13f160-1d4c-4337-a4b4-5d75a0c30eb0","Type":"ContainerDied","Data":"90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892"} Nov 02 06:42:27.812080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.811588 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a35950484ead1b8b1b60ea7982dfff3dbea87727950bbca5d53c8bfd922892" Nov 02 06:42:27.817597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.817571 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:42:27.822415 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:42:27.822375 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a78f19_1e8c_4433_a6cf_610311ad4c7d.slice/crio-0b47c40305fb24e4e7bc07543b1f70496a02230c30a476e454689b4875bb6cd4 WatchSource:0}: Error finding container 0b47c40305fb24e4e7bc07543b1f70496a02230c30a476e454689b4875bb6cd4: Status 404 returned error can't find the container with id 0b47c40305fb24e4e7bc07543b1f70496a02230c30a476e454689b4875bb6cd4 Nov 02 06:42:27.941515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:27.941485 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:42:27.946869 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:42:27.946844 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71732abe_2d44_49a7_a716_630dddf9b1bb.slice/crio-ec09f0afc941a34fa81acf21376aa946ae6f068150cff139f79f97da33d23dcd WatchSource:0}: Error finding container ec09f0afc941a34fa81acf21376aa946ae6f068150cff139f79f97da33d23dcd: Status 404 returned error can't find the container with id ec09f0afc941a34fa81acf21376aa946ae6f068150cff139f79f97da33d23dcd Nov 02 06:42:28.047376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.047351 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99"] Nov 02 06:42:28.050909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.050882 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac5fd2d-svz99"] Nov 02 06:42:28.146732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.146703 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:42:28.147472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.147205 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:42:28.147924 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.147902 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:42:28.180824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.180785 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647f2a2a-9a48-4f92-8236-6bafdc8e6456" path="/var/lib/kubelet/pods/647f2a2a-9a48-4f92-8236-6bafdc8e6456/volumes" Nov 02 06:42:28.820545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.820508 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" event={"ID":"71732abe-2d44-49a7-a716-630dddf9b1bb","Type":"ContainerStarted","Data":"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd"} Nov 02 06:42:28.821021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.820574 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" event={"ID":"71732abe-2d44-49a7-a716-630dddf9b1bb","Type":"ContainerStarted","Data":"ec09f0afc941a34fa81acf21376aa946ae6f068150cff139f79f97da33d23dcd"} Nov 02 06:42:28.821021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.820599 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:28.822212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.822182 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" event={"ID":"b3a78f19-1e8c-4433-a6cf-610311ad4c7d","Type":"ContainerStarted","Data":"981f6719d3827db0aa9df095de347e3b1773c070f54dc29bf79bea162679312f"} Nov 02 06:42:28.822317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.822233 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:28.822317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.822250 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" event={"ID":"b3a78f19-1e8c-4433-a6cf-610311ad4c7d","Type":"ContainerStarted","Data":"0b47c40305fb24e4e7bc07543b1f70496a02230c30a476e454689b4875bb6cd4"} Nov 02 06:42:28.932000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:28.931942 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" podStartSLOduration=2.9319253720000003 podStartE2EDuration="2.931925372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:28.928454752 +0000 UTC m=+1502.901789462" watchObservedRunningTime="2026-11-02 06:42:28.931925372 +0000 UTC m=+1502.905260059" Nov 02 06:42:29.045345 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:29.045108 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" podStartSLOduration=3.045087921 podStartE2EDuration="3.045087921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:29.035744108 +0000 UTC m=+1503.009078793" watchObservedRunningTime="2026-11-02 06:42:29.045087921 +0000 UTC m=+1503.018422607" Nov 02 06:42:32.047214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:32.047181 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs"] Nov 02 06:42:32.093612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:32.093585 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac85fe0-qsqfs"] Nov 02 06:42:32.181784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:32.181750 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a8fe3b-db2c-450f-91b2-43c3da75878c" path="/var/lib/kubelet/pods/98a8fe3b-db2c-450f-91b2-43c3da75878c/volumes" Nov 02 06:42:33.956613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:33.956584 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 06:42:33.957526 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:33.957502 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" containerName="revoke-keys" Nov 02 06:42:33.957526 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:33.957527 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" containerName="revoke-keys" Nov 02 06:42:33.957682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:33.957663 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" containerName="revoke-keys" Nov 02 06:42:33.961115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:33.961099 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.031281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.031248 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.031431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.031293 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.031649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.031620 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.031741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.031726 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnnf\" (UniqueName: \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.063674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.063643 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 06:42:34.104595 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.104570 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-3e96be-serving-cert\"" Nov 02 06:42:34.133420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.133396 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.133560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.133486 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnnf\" (UniqueName: \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.133613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.133584 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.133613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.133602 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.135030 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:42:34.134345 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-serving-cert: secret "maas-api-e2e-shared-a-w6-3e96be-serving-cert" not found Nov 02 06:42:34.135030 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:42:34.134448 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls podName:ed43483b-809e-4d2c-8fbc-e89f44d691b0 nodeName:}" failed. No retries permitted until 2026-11-02 06:42:34.634425837 +0000 UTC m=+1508.607760519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls") pod "maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" (UID: "ed43483b-809e-4d2c-8fbc-e89f44d691b0") : secret "maas-api-e2e-shared-a-w6-3e96be-serving-cert" not found Nov 02 06:42:34.135030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.134655 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.140030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.136951 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.191447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.191409 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnnf\" (UniqueName: \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.648309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.648274 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.650745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.650722 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:34.874129 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:34.874099 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:35.253311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:35.253225 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 06:42:35.258425 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:42:35.258394 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded43483b_809e_4d2c_8fbc_e89f44d691b0.slice/crio-0f4acf8dd5e71a8ebe9832105872faaa802b89a1473b241d530038447fc06172 WatchSource:0}: Error finding container 0f4acf8dd5e71a8ebe9832105872faaa802b89a1473b241d530038447fc06172: Status 404 returned error can't find the container with id 0f4acf8dd5e71a8ebe9832105872faaa802b89a1473b241d530038447fc06172 Nov 02 06:42:35.876894 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:35.876860 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" event={"ID":"ed43483b-809e-4d2c-8fbc-e89f44d691b0","Type":"ContainerStarted","Data":"0f4acf8dd5e71a8ebe9832105872faaa802b89a1473b241d530038447fc06172"} Nov 02 06:42:36.887032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:36.886975 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" event={"ID":"ed43483b-809e-4d2c-8fbc-e89f44d691b0","Type":"ContainerStarted","Data":"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc"} Nov 02 06:42:36.887418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:36.887122 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:42:36.996518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:36.996475 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" podStartSLOduration=3.075527896 podStartE2EDuration="3.996462746s" totalImagesPullingTime="920.93485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:36.994616163 +0000 UTC m=+1510.967950848" watchObservedRunningTime="2026-11-02 06:42:36.996462746 +0000 UTC m=+1510.969797429" Nov 02 06:42:37.045528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:37.045496 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx"] Nov 02 06:42:37.048927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:37.048901 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb239dfc-m59kx"] Nov 02 06:42:38.182534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:38.182501 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b44e68c-1fac-4bfa-bab5-e11c49e140f6" path="/var/lib/kubelet/pods/6b44e68c-1fac-4bfa-bab5-e11c49e140f6/volumes" Nov 02 06:42:38.560860 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:42:38.560741 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:42:39.833596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:39.833560 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:42:39.834198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:39.833638 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:42:42.904174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:42:42.904137 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 06:43:04.652255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.652175 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:43:04.652644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.652381 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" containerID="cri-o://08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0" gracePeriod=30 Nov 02 06:43:04.698277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.698250 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:43:04.698454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.698433 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" containerID="cri-o://0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0" gracePeriod=30 Nov 02 06:43:04.701743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.701721 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:43:04.701919 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.701899 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" containerID="cri-o://54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f" gracePeriod=30 Nov 02 06:43:04.745568 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.745538 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:43:04.745783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:04.745757 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" containerID="cri-o://111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687" gracePeriod=30 Nov 02 06:43:06.836338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:06.836302 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:43:08.579596 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:08.579349 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:43:08.804248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:08.804206 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:43:08.804431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:08.804278 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:43:10.058754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.058728 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:43:10.076157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.076126 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:43:10.084595 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.084565 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.084723 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.084625 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.084970 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.084944 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:10.085062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.084949 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.085062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085047 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.085175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085121 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.085244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085226 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.085366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085350 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085602 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085643 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.085698 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\") pod \"0ace3267-1856-40b6-90ff-c6f4c541bab4\" (UID: \"0ace3267-1856-40b6-90ff-c6f4c541bab4\") " Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086212 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086275 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086583 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086602 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086616 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace3267-1856-40b6-90ff-c6f4c541bab4-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086629 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.086711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.086690 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.088667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.088642 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:10.090852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.090827 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.093420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.093393 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "kube-api-access-6fw62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.094652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.094623 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token" pod "0ace3267-1856-40b6-90ff-c6f4c541bab4" (UID: "0ace3267-1856-40b6-90ff-c6f4c541bab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.130222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.130201 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:43:10.135490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.135470 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:43:10.177501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.177477 6273 generic.go:356] "Generic (PLEG): container finished" podID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerID="111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687" exitCode=0 Nov 02 06:43:10.177623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.177568 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" Nov 02 06:43:10.178967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.178943 6273 generic.go:356] "Generic (PLEG): container finished" podID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerID="0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0" exitCode=0 Nov 02 06:43:10.179089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.179050 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" Nov 02 06:43:10.180543 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.180523 6273 generic.go:356] "Generic (PLEG): container finished" podID="98f46032-62d2-4d9e-88be-16e246168c97" containerID="54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f" exitCode=0 Nov 02 06:43:10.180628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.180614 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" Nov 02 06:43:10.182043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.182013 6273 generic.go:356] "Generic (PLEG): container finished" podID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerID="08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0" exitCode=0 Nov 02 06:43:10.182130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.182111 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" Nov 02 06:43:10.186861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186842 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" event={"ID":"e951fc98-7e26-463d-9eb8-5fe78830aad2","Type":"ContainerDied","Data":"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687"} Nov 02 06:43:10.186955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186882 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q" event={"ID":"e951fc98-7e26-463d-9eb8-5fe78830aad2","Type":"ContainerDied","Data":"266b35a793c6b33242da4dcfaa47333f18dc289a3930ddf32a2f7536250ca643"} Nov 02 06:43:10.186955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186904 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" event={"ID":"7d87ffb4-0106-4de1-bc5b-ae552d514f7c","Type":"ContainerDied","Data":"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0"} Nov 02 06:43:10.186955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186920 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm" event={"ID":"7d87ffb4-0106-4de1-bc5b-ae552d514f7c","Type":"ContainerDied","Data":"5f3bda28e0e4cd214715cb65fa1761d9ba72bcb559598a38f43cb535269bcc4e"} Nov 02 06:43:10.186955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186934 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" event={"ID":"98f46032-62d2-4d9e-88be-16e246168c97","Type":"ContainerDied","Data":"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f"} Nov 02 06:43:10.186955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186951 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh" event={"ID":"98f46032-62d2-4d9e-88be-16e246168c97","Type":"ContainerDied","Data":"4fe8a5521f142ed9c423cb515eaff5514102f6834407e6452d17f4cc57b852fa"} Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186958 6273 scope.go:122] "RemoveContainer" containerID="111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687" Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186968 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" event={"ID":"0ace3267-1856-40b6-90ff-c6f4c541bab4","Type":"ContainerDied","Data":"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0"} Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.186999 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f" event={"ID":"0ace3267-1856-40b6-90ff-c6f4c541bab4","Type":"ContainerDied","Data":"9e362beb3cd652f7f4222adfdf638184bd0e5d4fb0b44cc2e1a736fabb3e22ce"} Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187007 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187054 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187082 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187174 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187210 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187250 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187283 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187313 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187352 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187385 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187518 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187548 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187620 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187659 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187705 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187796 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187829 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187862 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187891 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187893 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187926 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.187953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.187954 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188044 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188072 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188149 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188183 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\") pod \"e951fc98-7e26-463d-9eb8-5fe78830aad2\" (UID: \"e951fc98-7e26-463d-9eb8-5fe78830aad2\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188279 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\") pod \"98f46032-62d2-4d9e-88be-16e246168c97\" (UID: \"98f46032-62d2-4d9e-88be-16e246168c97\") " Nov 02 06:43:10.188516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.188312 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\") pod \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\" (UID: \"7d87ffb4-0106-4de1-bc5b-ae552d514f7c\") " Nov 02 06:43:10.189174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189156 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189181 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fw62\" (UniqueName: \"kubernetes.io/projected/0ace3267-1856-40b6-90ff-c6f4c541bab4-kube-api-access-6fw62\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189197 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189212 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189225 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e951fc98-7e26-463d-9eb8-5fe78830aad2-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189240 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace3267-1856-40b6-90ff-c6f4c541bab4-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.189596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189515 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.189725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189707 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.189781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189729 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.189892 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.189875 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.190117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.190098 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.190311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.190292 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.191472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.191378 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.191472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.191443 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:10.192171 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192096 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.192274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192261 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.192350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192308 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.192631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192592 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.192736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192684 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.192926 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192902 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.193059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192922 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:10.193059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.192964 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:10.193360 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.193341 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.193360 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.193355 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "kube-api-access-k9nx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.193514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.193391 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:10.193576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.193523 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.193576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.193540 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.195491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.195447 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "kube-api-access-h9f5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.195586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.195527 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.195586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.195547 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy" pod "98f46032-62d2-4d9e-88be-16e246168c97" (UID: "98f46032-62d2-4d9e-88be-16e246168c97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:10.196361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.196335 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo" pod "7d87ffb4-0106-4de1-bc5b-ae552d514f7c" (UID: "7d87ffb4-0106-4de1-bc5b-ae552d514f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:10.197522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.197501 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc" pod "e951fc98-7e26-463d-9eb8-5fe78830aad2" (UID: "e951fc98-7e26-463d-9eb8-5fe78830aad2"). InnerVolumeSpecName "kube-api-access-g82vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:10.202500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.202480 6273 scope.go:122] "RemoveContainer" containerID="111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687" Nov 02 06:43:10.202760 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:10.202739 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687\": container with ID starting with 111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687 not found: ID does not exist" containerID="111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687" Nov 02 06:43:10.202813 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.202767 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687"} err="failed to get container status \"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687\": rpc error: code = NotFound desc = could not find container \"111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687\": container with ID starting with 111e031309f788b98ea895b4ad1ae80c77686755af8c089d5e7df1546bf3b687 not found: ID does not exist" Nov 02 06:43:10.202813 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.202787 6273 scope.go:122] "RemoveContainer" containerID="0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0" Nov 02 06:43:10.211837 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.211820 6273 scope.go:122] "RemoveContainer" containerID="0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0" Nov 02 06:43:10.212141 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:10.212116 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0\": container with ID starting with 0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0 not found: ID does not exist" containerID="0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0" Nov 02 06:43:10.212243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.212147 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0"} err="failed to get container status \"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0\": rpc error: code = NotFound desc = could not find container \"0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0\": container with ID starting with 0258d40527cc1ce4aeb27e379cfce61c13478a6e276063938d25bfe0c70af8d0 not found: ID does not exist" Nov 02 06:43:10.212243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.212176 6273 scope.go:122] "RemoveContainer" containerID="54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f" Nov 02 06:43:10.223042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.223021 6273 scope.go:122] "RemoveContainer" containerID="54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f" Nov 02 06:43:10.223358 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:10.223338 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f\": container with ID starting with 54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f not found: ID does not exist" containerID="54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f" Nov 02 06:43:10.223435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.223364 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f"} err="failed to get container status \"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f\": rpc error: code = NotFound desc = could not find container \"54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f\": container with ID starting with 54b8285c854ce1b346c99e9221c6625650946232a01363028b9f283aabddfd6f not found: ID does not exist" Nov 02 06:43:10.223435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.223382 6273 scope.go:122] "RemoveContainer" containerID="08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0" Nov 02 06:43:10.234599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.234573 6273 scope.go:122] "RemoveContainer" containerID="08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0" Nov 02 06:43:10.234907 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:10.234884 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0\": container with ID starting with 08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0 not found: ID does not exist" containerID="08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0" Nov 02 06:43:10.235026 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.234914 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0"} err="failed to get container status \"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0\": rpc error: code = NotFound desc = could not find container \"08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0\": container with ID starting with 08e3bd49e0f7ba51fd1f3b37444c3cb81a64a45615b69c77aeed2e71e6e900b0 not found: ID does not exist" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290340 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290384 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290400 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9f5n\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-kube-api-access-h9f5n\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290416 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f46032-62d2-4d9e-88be-16e246168c97-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290426 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290434 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290443 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290456 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290468 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290480 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82vc\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-kube-api-access-g82vc\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290491 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f46032-62d2-4d9e-88be-16e246168c97-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290499 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290507 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290520 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290531 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290544 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290555 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290568 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290582 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.290588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290596 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290609 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f46032-62d2-4d9e-88be-16e246168c97-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290623 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nx4\" (UniqueName: \"kubernetes.io/projected/98f46032-62d2-4d9e-88be-16e246168c97-kube-api-access-k9nx4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290635 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290646 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290661 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d87ffb4-0106-4de1-bc5b-ae552d514f7c-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.291698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.290675 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e951fc98-7e26-463d-9eb8-5fe78830aad2-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:10.554745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.554683 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:43:10.559594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.559570 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-njh4f"] Nov 02 06:43:10.775350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.775284 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:43:10.783087 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.783061 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-82wvm"] Nov 02 06:43:10.887255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.887219 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:43:10.895077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:10.895045 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-hpnkh"] Nov 02 06:43:11.003770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:11.003734 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:43:11.006916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:11.006894 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-ntb5q"] Nov 02 06:43:12.185215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:12.185174 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" path="/var/lib/kubelet/pods/0ace3267-1856-40b6-90ff-c6f4c541bab4/volumes" Nov 02 06:43:12.185834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:12.185814 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" path="/var/lib/kubelet/pods/7d87ffb4-0106-4de1-bc5b-ae552d514f7c/volumes" Nov 02 06:43:12.186424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:12.186405 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f46032-62d2-4d9e-88be-16e246168c97" path="/var/lib/kubelet/pods/98f46032-62d2-4d9e-88be-16e246168c97/volumes" Nov 02 06:43:12.187551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:12.187526 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" path="/var/lib/kubelet/pods/e951fc98-7e26-463d-9eb8-5fe78830aad2/volumes" Nov 02 06:43:19.360727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.360687 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:43:19.361497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361465 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" Nov 02 06:43:19.361497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361489 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361541 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361551 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361627 6273 cpu_manager.go:414] "containerMap: removing container" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361636 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361669 6273 cpu_manager.go:414] "containerMap: removing container" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" Nov 02 06:43:19.361707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.361676 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" Nov 02 06:43:19.362428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.362404 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ace3267-1856-40b6-90ff-c6f4c541bab4" containerName="istio-proxy" Nov 02 06:43:19.362564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.362439 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f46032-62d2-4d9e-88be-16e246168c97" containerName="istio-proxy" Nov 02 06:43:19.362564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.362454 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d87ffb4-0106-4de1-bc5b-ae552d514f7c" containerName="istio-proxy" Nov 02 06:43:19.362564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.362470 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e951fc98-7e26-463d-9eb8-5fe78830aad2" containerName="istio-proxy" Nov 02 06:43:19.366383 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.366362 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.420518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.420480 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:43:19.463009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.462019 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-08bbf9\"" Nov 02 06:43:19.463009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.462195 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-08bbf9-dockercfg-mrlfc\"" Nov 02 06:43:19.516299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.516261 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:43:19.521495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.521464 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.544170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.544135 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4dg\" (UniqueName: \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.544383 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.544364 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.600039 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.599748 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:43:19.668088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.666071 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4vt\" (UniqueName: \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.668088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.666311 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4dg\" (UniqueName: \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.668088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.666780 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.668088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.666954 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.668088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.667851 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.759897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.759850 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4dg\" (UniqueName: \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\") pod \"payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:19.769524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.769489 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4vt\" (UniqueName: \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.769738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.769707 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.770499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.770479 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.824881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.824849 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4vt\" (UniqueName: \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\") pod \"payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.842240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.842212 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:19.991354 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:19.987362 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:20.231121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:20.231043 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:43:20.236591 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:43:20.236564 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111b1c1f_09c7_482e_b915_2a1cdb967a70.slice/crio-1f2ecd7176ad93e7098b32ebb529cddf98f6f80543dc8d8f500c179d5c818c7a WatchSource:0}: Error finding container 1f2ecd7176ad93e7098b32ebb529cddf98f6f80543dc8d8f500c179d5c818c7a: Status 404 returned error can't find the container with id 1f2ecd7176ad93e7098b32ebb529cddf98f6f80543dc8d8f500c179d5c818c7a Nov 02 06:43:20.272413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:20.272386 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" event={"ID":"111b1c1f-09c7-482e-b915-2a1cdb967a70","Type":"ContainerStarted","Data":"1f2ecd7176ad93e7098b32ebb529cddf98f6f80543dc8d8f500c179d5c818c7a"} Nov 02 06:43:20.427332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:20.427261 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:43:20.437511 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:43:20.437478 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089de147_27e1_467f_bfbd_753ab389ca0e.slice/crio-6772b12451d91bceadb440f898a64d04daa1e26d85cf02bd6be1dcf4ff4d63e1 WatchSource:0}: Error finding container 6772b12451d91bceadb440f898a64d04daa1e26d85cf02bd6be1dcf4ff4d63e1: Status 404 returned error can't find the container with id 6772b12451d91bceadb440f898a64d04daa1e26d85cf02bd6be1dcf4ff4d63e1 Nov 02 06:43:20.488404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:20.488337 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2"] Nov 02 06:43:20.491771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:20.491740 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9fdd1c-pxwh2"] Nov 02 06:43:21.045757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.045712 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:43:21.046020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.045975 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" containerID="cri-o://846e36c0671e50fe2087c2017fbeb4eb0ce4fdee8ad6239cbfd05dcd40d7368b" gracePeriod=30 Nov 02 06:43:21.299269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.299151 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" event={"ID":"111b1c1f-09c7-482e-b915-2a1cdb967a70","Type":"ContainerStarted","Data":"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b"} Nov 02 06:43:21.301150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.301112 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:21.302400 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.302367 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" event={"ID":"089de147-27e1-467f-bfbd-753ab389ca0e","Type":"ContainerStarted","Data":"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7"} Nov 02 06:43:21.302524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.302413 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" event={"ID":"089de147-27e1-467f-bfbd-753ab389ca0e","Type":"ContainerStarted","Data":"6772b12451d91bceadb440f898a64d04daa1e26d85cf02bd6be1dcf4ff4d63e1"} Nov 02 06:43:21.302524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.302450 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:21.429676 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.429628 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" podStartSLOduration=2.429612543 podStartE2EDuration="2.429612543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:21.427953536 +0000 UTC m=+1555.401288230" watchObservedRunningTime="2026-11-02 06:43:21.429612543 +0000 UTC m=+1555.402947274" Nov 02 06:43:21.539614 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:21.539558 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" podStartSLOduration=2.539546032 podStartE2EDuration="2.539546032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:21.537947122 +0000 UTC m=+1555.511281809" watchObservedRunningTime="2026-11-02 06:43:21.539546032 +0000 UTC m=+1555.512880714" Nov 02 06:43:22.183636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:22.183596 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c13f160-1d4c-4337-a4b4-5d75a0c30eb0" path="/var/lib/kubelet/pods/1c13f160-1d4c-4337-a4b4-5d75a0c30eb0/volumes" Nov 02 06:43:25.110150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.110113 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:43:25.116164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.116140 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.168900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.168871 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:43:25.193606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.193575 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjn85\" (UniqueName: \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.193794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.193768 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.193887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.193871 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.193950 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.193939 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.210191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.210162 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-29abe7-serving-cert\"" Nov 02 06:43:25.258976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.258935 6273 patch_prober.go:29] interesting pod/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:43:25.259168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.259106 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:43:25.295328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.295295 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.295488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.295380 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.295554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.295494 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.295554 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:25.295521 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-serving-cert: secret "maas-api-e2e-shared-b-w0-29abe7-serving-cert" not found Nov 02 06:43:25.295658 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:25.295594 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls podName:506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3 nodeName:}" failed. No retries permitted until 2026-11-02 06:43:25.795573107 +0000 UTC m=+1559.768907781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls") pod "maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" (UID: "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3") : secret "maas-api-e2e-shared-b-w0-29abe7-serving-cert" not found Nov 02 06:43:25.295772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.295752 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjn85\" (UniqueName: \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.296214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.296189 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.297786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.297766 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.352180 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.352143 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjn85\" (UniqueName: \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.809229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.808402 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:25.812967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:25.812934 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:26.035816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.035782 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:26.344267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.344240 6273 generic.go:356] "Generic (PLEG): container finished" podID="a074febf-476f-4962-92f7-8bdc3ecac195" containerID="846e36c0671e50fe2087c2017fbeb4eb0ce4fdee8ad6239cbfd05dcd40d7368b" exitCode=0 Nov 02 06:43:26.344603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.344312 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerDied","Data":"846e36c0671e50fe2087c2017fbeb4eb0ce4fdee8ad6239cbfd05dcd40d7368b"} Nov 02 06:43:26.344603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.344365 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" event={"ID":"a074febf-476f-4962-92f7-8bdc3ecac195","Type":"ContainerDied","Data":"47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5"} Nov 02 06:43:26.344603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.344382 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47996618ae0652328cf9cff5fad0179449ec72e1dbef3f686f7f976c741e46a5" Nov 02 06:43:26.344603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.344396 6273 scope.go:122] "RemoveContainer" containerID="c743ca3c5aa0ae00fcb5fbb2dbdbcb96f7be7e709da515554edfd419db30bb5b" Nov 02 06:43:26.349049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.349029 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:43:26.419717 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:43:26.419688 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506bf7ec_f8f8_42a6_9fb3_b5cd9bcef4d3.slice/crio-074a81b70f6f9c740dcb95cba720b785ded1cf6fb1d3a51081537de54f0df301 WatchSource:0}: Error finding container 074a81b70f6f9c740dcb95cba720b785ded1cf6fb1d3a51081537de54f0df301: Status 404 returned error can't find the container with id 074a81b70f6f9c740dcb95cba720b785ded1cf6fb1d3a51081537de54f0df301 Nov 02 06:43:26.448160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.448135 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:43:26.521528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.521501 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.521875 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.521797 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522035 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522125 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522164 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522269 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522304 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522326 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522357 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522400 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522466 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\") pod \"a074febf-476f-4962-92f7-8bdc3ecac195\" (UID: \"a074febf-476f-4962-92f7-8bdc3ecac195\") " Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.522709 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.523386 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.523410 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.523611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.523427 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074febf-476f-4962-92f7-8bdc3ecac195-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.524314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.523675 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:26.524477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.524445 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:26.526193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.526166 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "kube-api-access-z8v86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:26.527563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.527538 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:26.527680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.527647 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:26.527949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.527930 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy" pod "a074febf-476f-4962-92f7-8bdc3ecac195" (UID: "a074febf-476f-4962-92f7-8bdc3ecac195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624630 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074febf-476f-4962-92f7-8bdc3ecac195-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624656 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624666 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624675 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624684 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v86\" (UniqueName: \"kubernetes.io/projected/a074febf-476f-4962-92f7-8bdc3ecac195-kube-api-access-z8v86\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.624699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.624693 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074febf-476f-4962-92f7-8bdc3ecac195-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:26.650937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.650908 6273 scope.go:122] "RemoveContainer" containerID="e9f511f936512d6e81b7eb3fb13335b455aaec761e8f9450114037936200cdbf" Nov 02 06:43:26.661302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.661276 6273 scope.go:122] "RemoveContainer" containerID="7c233b460ec234c64a9d08d877962e25abf47df0caf957aed4dac28a8ddc6dc1" Nov 02 06:43:26.670073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.670054 6273 scope.go:122] "RemoveContainer" containerID="a8f6b0855e4a19d19a4eb250e6926a12860612b7f96984a82350b1cad3ed68cb" Nov 02 06:43:26.679918 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.679898 6273 scope.go:122] "RemoveContainer" containerID="69f81c20a4c09c8a374132a2bbf50ba375cd6708c0cf913b2a598000760ce7a7" Nov 02 06:43:26.688462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.688439 6273 scope.go:122] "RemoveContainer" containerID="c9ed639faacfb285d96191c2a75caaa8725f1c101b6fb13e3e088a99b87fdcd9" Nov 02 06:43:26.697355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.697333 6273 scope.go:122] "RemoveContainer" containerID="60a4ffda0c8d165918bad4fa2d3f008425c23e2e03c6b79ac0507bc6039d0dff" Nov 02 06:43:26.705705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:26.705686 6273 scope.go:122] "RemoveContainer" containerID="a0b4f6f60ed65e0f2e1786c2098640578fcf53b11700ce64aad504c2b9b1d8d5" Nov 02 06:43:27.353960 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:27.353934 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k" Nov 02 06:43:27.355566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:27.355540 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" event={"ID":"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3","Type":"ContainerStarted","Data":"074a81b70f6f9c740dcb95cba720b785ded1cf6fb1d3a51081537de54f0df301"} Nov 02 06:43:27.592684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:27.592606 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:43:27.599713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:27.599683 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc9fdd1c-openshift-default-867b946cd9-sbk4k"] Nov 02 06:43:28.193828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:28.193795 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" path="/var/lib/kubelet/pods/a074febf-476f-4962-92f7-8bdc3ecac195/volumes" Nov 02 06:43:28.379603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:28.379548 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" event={"ID":"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3","Type":"ContainerStarted","Data":"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd"} Nov 02 06:43:28.380806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:28.380774 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:28.495151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:28.495053 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" podStartSLOduration=2.609670679 podStartE2EDuration="3.495040209s" totalImagesPullingTime="885.36953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:28.492763416 +0000 UTC m=+1562.466098100" watchObservedRunningTime="2026-11-02 06:43:28.495040209 +0000 UTC m=+1562.468374929" Nov 02 06:43:31.027499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.027468 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:43:31.028189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.028172 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:43:31.028189 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.028187 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:43:31.028336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.028197 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:43:31.028336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.028202 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:43:31.028336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.028334 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:43:31.032510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.032490 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.086182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.086150 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:43:31.134012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.116039 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znf2t\" (UniqueName: \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.134012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.116217 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.134012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.128069 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-70898f\"" Nov 02 06:43:31.134012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.128196 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-70898f-dockercfg-4t9gt\"" Nov 02 06:43:31.218019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.217734 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znf2t\" (UniqueName: \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.218457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.218430 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.228205 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.219863 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.273895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.273869 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znf2t\" (UniqueName: \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\") pod \"payload-processing-e2e-api-a-70898f-78564c9687-rw28r\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.353800 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.353707 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:31.785040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:31.785005 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:43:31.791683 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:43:31.791659 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d3b437_1421_4c3f_9d17_ed2d00649991.slice/crio-c66fb97f334003fa4e8cdfa06c14d528d86f455f542b43e3ad7f93482f89e416 WatchSource:0}: Error finding container c66fb97f334003fa4e8cdfa06c14d528d86f455f542b43e3ad7f93482f89e416: Status 404 returned error can't find the container with id c66fb97f334003fa4e8cdfa06c14d528d86f455f542b43e3ad7f93482f89e416 Nov 02 06:43:32.312515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:32.312488 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:43:32.418337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:32.418299 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" event={"ID":"97d3b437-1421-4c3f-9d17-ed2d00649991","Type":"ContainerStarted","Data":"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034"} Nov 02 06:43:32.418516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:32.418391 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" event={"ID":"97d3b437-1421-4c3f-9d17-ed2d00649991","Type":"ContainerStarted","Data":"c66fb97f334003fa4e8cdfa06c14d528d86f455f542b43e3ad7f93482f89e416"} Nov 02 06:43:32.418516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:32.418424 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:33.322726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:33.322693 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:43:33.432103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:33.432054 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" podStartSLOduration=3.4320385 podStartE2EDuration="3.4320385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:32.528894264 +0000 UTC m=+1566.502228950" watchObservedRunningTime="2026-11-02 06:43:33.4320385 +0000 UTC m=+1567.405373182" Nov 02 06:43:35.414588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:35.414557 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:43:38.553423 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:38.553385 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:43:43.428522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:43.428489 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:43:46.981198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:46.981163 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:43:48.906170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.906114 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:43:48.906592 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.906331 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" containerID="cri-o://ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb" gracePeriod=30 Nov 02 06:43:48.952770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.952743 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:43:48.952982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.952962 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" containerID="cri-o://c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e" gracePeriod=30 Nov 02 06:43:48.956000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.955959 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:43:48.956163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.956141 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" containerID="cri-o://b7605c27792e3515a40ecfe3c42bf619272647872cfb25ccafa3b02492b91225" gracePeriod=30 Nov 02 06:43:48.998305 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.998280 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:43:48.998454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:48.998437 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" containerID="cri-o://cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15" gracePeriod=30 Nov 02 06:43:49.388200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:49.388119 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:43:49.388376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:49.388296 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" containerID="cri-o://80040da937316436666df9c8a47f102f936d7a1b3195e6130706776306fed739" gracePeriod=30 Nov 02 06:43:49.672565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:49.672527 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:43:49.672784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:49.672719 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" containerID="cri-o://3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f" gracePeriod=30 Nov 02 06:43:54.008042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.001204 6273 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 06:43:54.008042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.001266 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 06:43:54.256270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.256216 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:43:54.350921 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.350890 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:43:54.362435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.362410 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:43:54.378504 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378470 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.378652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378559 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.378721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378652 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.378786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378721 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.378919 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378900 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.378980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378938 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.379059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.378998 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.379136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.379119 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.379199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.379183 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\") pod \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\" (UID: \"3286ad65-78af-4cda-b1fa-aae9ee36ed56\") " Nov 02 06:43:54.379577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.379552 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.380004 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.379965 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.380365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.380344 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.381603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.381336 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.382325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.382296 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.382588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.382564 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.383886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.383853 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.384469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.384437 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "kube-api-access-s6ggn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.385196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.385151 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.389182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.389159 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token" pod "3286ad65-78af-4cda-b1fa-aae9ee36ed56" (UID: "3286ad65-78af-4cda-b1fa-aae9ee36ed56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.481720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.481613 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.482251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482211 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.482388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482267 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.482388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482304 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.482388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482330 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.482661 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482572 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.482720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482696 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.482780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482740 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.482780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482768 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.482887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482811 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.482887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482846 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.483031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482905 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.483031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.482952 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.483198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483110 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.483198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483125 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.483198 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483173 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.483356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483225 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.483408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483366 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.483408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483365 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.483513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483435 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.483513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483487 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\") pod \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\" (UID: \"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b\") " Nov 02 06:43:54.483609 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483536 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.483609 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483555 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\") pod \"9d7073f1-06af-4968-b7d3-c138d240ce9b\" (UID: \"9d7073f1-06af-4968-b7d3-c138d240ce9b\") " Nov 02 06:43:54.487640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.483738 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.487640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.484388 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.487640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.486619 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.487640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.487046 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.487640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.487540 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.488700 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.488663 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.489983 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.489925 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "kube-api-access-dv2k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.491089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.491046 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.491946 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.491924 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.492118 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.491959 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.492435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.492397 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "kube-api-access-txj8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.493069 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.493034 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token" pod "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" (UID: "5cac6543-e4b4-4b5f-aa53-3aebb7c8270b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.494243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494214 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494246 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494262 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ggn\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-kube-api-access-s6ggn\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494276 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494288 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2k2\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-kube-api-access-dv2k2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494302 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494314 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494327 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494339 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494351 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.494367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494363 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494375 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txj8b\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-kube-api-access-txj8b\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494388 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494399 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494410 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494422 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3286ad65-78af-4cda-b1fa-aae9ee36ed56-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494433 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494445 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494456 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3286ad65-78af-4cda-b1fa-aae9ee36ed56-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494469 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494483 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494496 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494507 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494522 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7073f1-06af-4968-b7d3-c138d240ce9b-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494534 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.495341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.494639 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo" pod "9d7073f1-06af-4968-b7d3-c138d240ce9b" (UID: "9d7073f1-06af-4968-b7d3-c138d240ce9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.622252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.619585 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7073f1-06af-4968-b7d3-c138d240ce9b-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.623961 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.623925 6273 generic.go:356] "Generic (PLEG): container finished" podID="861873a0-3f3a-4bbf-81eb-28c412173941" containerID="b7605c27792e3515a40ecfe3c42bf619272647872cfb25ccafa3b02492b91225" exitCode=0 Nov 02 06:43:54.624244 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.624218 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" event={"ID":"861873a0-3f3a-4bbf-81eb-28c412173941","Type":"ContainerDied","Data":"b7605c27792e3515a40ecfe3c42bf619272647872cfb25ccafa3b02492b91225"} Nov 02 06:43:54.634041 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.634007 6273 generic.go:356] "Generic (PLEG): container finished" podID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerID="ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb" exitCode=0 Nov 02 06:43:54.636119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.635406 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" Nov 02 06:43:54.639055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.636435 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerDied","Data":"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb"} Nov 02 06:43:54.639055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.636505 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w" event={"ID":"3286ad65-78af-4cda-b1fa-aae9ee36ed56","Type":"ContainerDied","Data":"a08fd5491a1a2214f8c8ec599c711514d58b984f15a6f32b493de3679ca26e7a"} Nov 02 06:43:54.639055 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.636544 6273 scope.go:122] "RemoveContainer" containerID="ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb" Nov 02 06:43:54.663311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.662150 6273 generic.go:356] "Generic (PLEG): container finished" podID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerID="c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e" exitCode=0 Nov 02 06:43:54.663311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.662428 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" event={"ID":"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b","Type":"ContainerDied","Data":"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e"} Nov 02 06:43:54.663311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.662478 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" event={"ID":"5cac6543-e4b4-4b5f-aa53-3aebb7c8270b","Type":"ContainerDied","Data":"fac7f87863fa570b6e3271168530da03c2686c5e5d28a9fe4e953610c019f039"} Nov 02 06:43:54.663311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.662614 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66" Nov 02 06:43:54.667560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.665523 6273 scope.go:122] "RemoveContainer" containerID="04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6" Nov 02 06:43:54.667560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.665775 6273 generic.go:356] "Generic (PLEG): container finished" podID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerID="80040da937316436666df9c8a47f102f936d7a1b3195e6130706776306fed739" exitCode=0 Nov 02 06:43:54.667560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.666017 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" event={"ID":"2454edcd-0643-4d9c-9755-3d23403c47d3","Type":"ContainerDied","Data":"80040da937316436666df9c8a47f102f936d7a1b3195e6130706776306fed739"} Nov 02 06:43:54.669333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.669301 6273 generic.go:356] "Generic (PLEG): container finished" podID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerID="cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15" exitCode=0 Nov 02 06:43:54.669450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.669367 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" event={"ID":"9d7073f1-06af-4968-b7d3-c138d240ce9b","Type":"ContainerDied","Data":"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15"} Nov 02 06:43:54.669450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.669411 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" event={"ID":"9d7073f1-06af-4968-b7d3-c138d240ce9b","Type":"ContainerDied","Data":"641455c9db6d7aa8a50ebd7fe330112c08a1314d0aaa6c779a2982a65ba2a572"} Nov 02 06:43:54.669574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.669528 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw" Nov 02 06:43:54.683580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.683547 6273 scope.go:122] "RemoveContainer" containerID="ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb" Nov 02 06:43:54.684372 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:54.684347 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb\": container with ID starting with ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb not found: ID does not exist" containerID="ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb" Nov 02 06:43:54.684581 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.684555 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb"} err="failed to get container status \"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb\": rpc error: code = NotFound desc = could not find container \"ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb\": container with ID starting with ba8581208cc4b00200a93b70ddbbc1033446c7409b4896d49212bb46891dc8bb not found: ID does not exist" Nov 02 06:43:54.684739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.684723 6273 scope.go:122] "RemoveContainer" containerID="04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6" Nov 02 06:43:54.685353 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:54.685189 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6\": container with ID starting with 04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6 not found: ID does not exist" containerID="04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6" Nov 02 06:43:54.685353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.685228 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6"} err="failed to get container status \"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6\": rpc error: code = NotFound desc = could not find container \"04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6\": container with ID starting with 04f82c199436a910616181e8ca51191c54c768a31af795a018999a6e15b7f9a6 not found: ID does not exist" Nov 02 06:43:54.685353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.685251 6273 scope.go:122] "RemoveContainer" containerID="c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e" Nov 02 06:43:54.702033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.701965 6273 scope.go:122] "RemoveContainer" containerID="c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e" Nov 02 06:43:54.702383 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:54.702357 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e\": container with ID starting with c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e not found: ID does not exist" containerID="c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e" Nov 02 06:43:54.702476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.702394 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e"} err="failed to get container status \"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e\": rpc error: code = NotFound desc = could not find container \"c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e\": container with ID starting with c93d63458eaa1462683d4ef410176ae112290d1d0c039385463f8d28e4862e8e not found: ID does not exist" Nov 02 06:43:54.702476 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.702416 6273 scope.go:122] "RemoveContainer" containerID="cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15" Nov 02 06:43:54.718902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.718867 6273 scope.go:122] "RemoveContainer" containerID="cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15" Nov 02 06:43:54.719462 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:54.719437 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15\": container with ID starting with cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15 not found: ID does not exist" containerID="cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15" Nov 02 06:43:54.719548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.719469 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15"} err="failed to get container status \"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15\": rpc error: code = NotFound desc = could not find container \"cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15\": container with ID starting with cc7c0b91b9ec63dda952b1910e514a2d427b75f5a17862a44c4bcd2eec55fd15 not found: ID does not exist" Nov 02 06:43:54.765686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.765612 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:43:54.793312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.793287 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:43:54.805182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.805057 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:43:54.808395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.808372 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-bvp66"] Nov 02 06:43:54.911557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.911522 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:43:54.923227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923193 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923267 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923332 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923392 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923365 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.923570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923397 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.923570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923485 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923538 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923586 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.923730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923698 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.923835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923754 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.923835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923790 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-mwp9w"] Nov 02 06:43:54.923835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923797 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.924006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923887 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.924006 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.923970 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.924121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924022 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.924121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924064 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.924121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924085 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.924121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924117 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.924382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924183 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\") pod \"2454edcd-0643-4d9c-9755-3d23403c47d3\" (UID: \"2454edcd-0643-4d9c-9755-3d23403c47d3\") " Nov 02 06:43:54.924382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.924222 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\") pod \"861873a0-3f3a-4bbf-81eb-28c412173941\" (UID: \"861873a0-3f3a-4bbf-81eb-28c412173941\") " Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.926419 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.926783 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.927094 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.927322 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.927611 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.928021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.927849 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.931044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.931018 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.931337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.931304 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.931772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.931519 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.932103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.932083 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.938860 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.938837 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.939065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.938948 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.939143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.939066 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.939206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.939192 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.939288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.939207 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "kube-api-access-q927l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.940842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.940806 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294" pod "2454edcd-0643-4d9c-9755-3d23403c47d3" (UID: "2454edcd-0643-4d9c-9755-3d23403c47d3"). InnerVolumeSpecName "kube-api-access-j4294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.941974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:54.941941 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy" pod "861873a0-3f3a-4bbf-81eb-28c412173941" (UID: "861873a0-3f3a-4bbf-81eb-28c412173941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.003968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.003935 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:43:55.026610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026536 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026624 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026678 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026720 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026842 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.026877 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027040 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027046 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.027097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027098 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027541 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027141 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027541 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027190 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\") pod \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\" (UID: \"362133a2-bb5b-4d12-a639-db8ab15ee1a8\") " Nov 02 06:43:55.027541 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027095 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:55.027541 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027458 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.027750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027630 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.027867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027848 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.027928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027870 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2454edcd-0643-4d9c-9755-3d23403c47d3-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.027928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027884 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.027928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027901 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.027928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027915 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861873a0-3f3a-4bbf-81eb-28c412173941-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027928 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q927l\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-kube-api-access-q927l\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027942 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861873a0-3f3a-4bbf-81eb-28c412173941-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027955 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027968 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.027980 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028008 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028021 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861873a0-3f3a-4bbf-81eb-28c412173941-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028034 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028048 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028060 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028073 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028086 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028099 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861873a0-3f3a-4bbf-81eb-28c412173941-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028111 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028124 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4294\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-kube-api-access-j4294\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028137 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2454edcd-0643-4d9c-9755-3d23403c47d3-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028150 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.028173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.028163 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2454edcd-0643-4d9c-9755-3d23403c47d3-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.029337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.029315 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "kube-api-access-m5874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:55.029487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.029466 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:55.029552 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.029472 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:55.030607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.030578 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy" pod "362133a2-bb5b-4d12-a639-db8ab15ee1a8" (UID: "362133a2-bb5b-4d12-a639-db8ab15ee1a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.075201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.075152 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:43:55.079821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.079791 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-b2ldw"] Nov 02 06:43:55.130024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.129974 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.130024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.130024 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5874\" (UniqueName: \"kubernetes.io/projected/362133a2-bb5b-4d12-a639-db8ab15ee1a8-kube-api-access-m5874\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.130215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.130042 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.130215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.130057 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362133a2-bb5b-4d12-a639-db8ab15ee1a8-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.677856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.677817 6273 generic.go:356] "Generic (PLEG): container finished" podID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerID="3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f" exitCode=0 Nov 02 06:43:55.678097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.677871 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" Nov 02 06:43:55.678097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.677899 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" event={"ID":"362133a2-bb5b-4d12-a639-db8ab15ee1a8","Type":"ContainerDied","Data":"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f"} Nov 02 06:43:55.678097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.677948 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" event={"ID":"362133a2-bb5b-4d12-a639-db8ab15ee1a8","Type":"ContainerDied","Data":"88018d269d4d071c15be752c5733f39326679c8e2f1e0a906aa8a2714dd13d9d"} Nov 02 06:43:55.678097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.677971 6273 scope.go:122] "RemoveContainer" containerID="3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f" Nov 02 06:43:55.679655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.679635 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" Nov 02 06:43:55.679824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.679703 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc" event={"ID":"2454edcd-0643-4d9c-9755-3d23403c47d3","Type":"ContainerDied","Data":"d37ad6ff26c1570b6c5aa58bb9f01789caa12d780ddb90db375dffc68bca2435"} Nov 02 06:43:55.682898 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.682877 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" Nov 02 06:43:55.683028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.682890 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4" event={"ID":"861873a0-3f3a-4bbf-81eb-28c412173941","Type":"ContainerDied","Data":"673e127859d9e3b2230c396120a00e0c8ac4f32fbe8a4b385640634142c04054"} Nov 02 06:43:55.692664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.692635 6273 scope.go:122] "RemoveContainer" containerID="3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f" Nov 02 06:43:55.692980 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:43:55.692959 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f\": container with ID starting with 3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f not found: ID does not exist" containerID="3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f" Nov 02 06:43:55.693089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.693030 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f"} err="failed to get container status \"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f\": rpc error: code = NotFound desc = could not find container \"3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f\": container with ID starting with 3e50a04e89ab7d054868bc8c7996c5f0330c46d493b9e78f845577f944084b5f not found: ID does not exist" Nov 02 06:43:55.693089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.693057 6273 scope.go:122] "RemoveContainer" containerID="80040da937316436666df9c8a47f102f936d7a1b3195e6130706776306fed739" Nov 02 06:43:55.702698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.702681 6273 scope.go:122] "RemoveContainer" containerID="b7605c27792e3515a40ecfe3c42bf619272647872cfb25ccafa3b02492b91225" Nov 02 06:43:55.834433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.834402 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:43:55.848191 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.848164 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-5gcqc"] Nov 02 06:43:55.926647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.926617 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:43:55.926783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.926674 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:43:55.956494 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.956415 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:43:55.962614 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:55.962587 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-qssp4"] Nov 02 06:43:56.065430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.065397 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:43:56.070781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.070758 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-22cpg"] Nov 02 06:43:56.183202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.183171 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" path="/var/lib/kubelet/pods/2454edcd-0643-4d9c-9755-3d23403c47d3/volumes" Nov 02 06:43:56.183754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.183740 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" path="/var/lib/kubelet/pods/3286ad65-78af-4cda-b1fa-aae9ee36ed56/volumes" Nov 02 06:43:56.184281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.184266 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" path="/var/lib/kubelet/pods/362133a2-bb5b-4d12-a639-db8ab15ee1a8/volumes" Nov 02 06:43:56.185193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.185170 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" path="/var/lib/kubelet/pods/5cac6543-e4b4-4b5f-aa53-3aebb7c8270b/volumes" Nov 02 06:43:56.185827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.185788 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" path="/var/lib/kubelet/pods/861873a0-3f3a-4bbf-81eb-28c412173941/volumes" Nov 02 06:43:56.186319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:43:56.186298 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" path="/var/lib/kubelet/pods/9d7073f1-06af-4968-b7d3-c138d240ce9b/volumes" Nov 02 06:44:02.315296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315263 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 06:44:02.315909 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315895 6273 cpu_manager.go:414] "containerMap: removing container" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315911 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315927 6273 cpu_manager.go:414] "containerMap: removing container" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315932 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315944 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315949 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315963 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.315967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.315967 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316000 6273 cpu_manager.go:414] "containerMap: removing container" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316006 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316021 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316026 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316119 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="362133a2-bb5b-4d12-a639-db8ab15ee1a8" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316128 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cac6543-e4b4-4b5f-aa53-3aebb7c8270b" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316136 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316143 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a074febf-476f-4962-92f7-8bdc3ecac195" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316149 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="861873a0-3f3a-4bbf-81eb-28c412173941" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316155 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2454edcd-0643-4d9c-9755-3d23403c47d3" containerName="istio-proxy" Nov 02 06:44:02.316203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.316164 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7073f1-06af-4968-b7d3-c138d240ce9b" containerName="istio-proxy" Nov 02 06:44:02.321783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.321764 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.374709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.374663 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 06:44:02.412009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.411117 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-6e2657\"" Nov 02 06:44:02.437590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.437546 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.438084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.438054 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktgn\" (UniqueName: \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.454569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.454535 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-6e2657-dockercfg-kzlxt\"" Nov 02 06:44:02.499751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.499721 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 06:44:02.502145 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.501237 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.502329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.502310 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.502653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.502638 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3286ad65-78af-4cda-b1fa-aae9ee36ed56" containerName="istio-proxy" Nov 02 06:44:02.507117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.507086 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.540040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.539095 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.540290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.540259 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktgn\" (UniqueName: \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.540447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.540423 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvprz\" (UniqueName: \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.540758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.540730 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.541643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.541617 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.552241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.552208 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 06:44:02.595869 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.595785 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktgn\" (UniqueName: \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\") pod \"payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.636237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.636197 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:02.642674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.642647 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvprz\" (UniqueName: \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.642780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.642760 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.643497 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.643476 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.699616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.699588 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvprz\" (UniqueName: \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\") pod \"payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:02.838503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:02.838460 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:03.022834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.022800 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 06:44:03.028338 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:44:03.028316 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe0da50_df40_4fc7_a387_8734f87c2080.slice/crio-0c41a7941f0988570983e2326bac04625ed74d46d0b470987e0a3bd975938b4b WatchSource:0}: Error finding container 0c41a7941f0988570983e2326bac04625ed74d46d0b470987e0a3bd975938b4b: Status 404 returned error can't find the container with id 0c41a7941f0988570983e2326bac04625ed74d46d0b470987e0a3bd975938b4b Nov 02 06:44:03.220682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.220652 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 06:44:03.226390 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:44:03.226364 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842d153e_144e_4f7f_ad3c_03360e8e2403.slice/crio-490f07e3df9ea0e51f2cffb4403b61bfc495db6a7d2c31a468fd56935cf12767 WatchSource:0}: Error finding container 490f07e3df9ea0e51f2cffb4403b61bfc495db6a7d2c31a468fd56935cf12767: Status 404 returned error can't find the container with id 490f07e3df9ea0e51f2cffb4403b61bfc495db6a7d2c31a468fd56935cf12767 Nov 02 06:44:03.747934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.747884 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" event={"ID":"4fe0da50-df40-4fc7-a387-8734f87c2080","Type":"ContainerStarted","Data":"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4"} Nov 02 06:44:03.748356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.747951 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:03.748356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.747969 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" event={"ID":"4fe0da50-df40-4fc7-a387-8734f87c2080","Type":"ContainerStarted","Data":"0c41a7941f0988570983e2326bac04625ed74d46d0b470987e0a3bd975938b4b"} Nov 02 06:44:03.749904 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.749872 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" event={"ID":"842d153e-144e-4f7f-ad3c-03360e8e2403","Type":"ContainerStarted","Data":"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b"} Nov 02 06:44:03.750075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.749918 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" event={"ID":"842d153e-144e-4f7f-ad3c-03360e8e2403","Type":"ContainerStarted","Data":"490f07e3df9ea0e51f2cffb4403b61bfc495db6a7d2c31a468fd56935cf12767"} Nov 02 06:44:03.750075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.749939 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:03.857747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.857704 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" podStartSLOduration=1.8576909499999998 podStartE2EDuration="1.85769095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:03.854726747 +0000 UTC m=+1597.828061433" watchObservedRunningTime="2026-11-02 06:44:03.85769095 +0000 UTC m=+1597.831025633" Nov 02 06:44:03.982328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:03.967139 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" podStartSLOduration=1.9671211180000001 podStartE2EDuration="1.967121118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:03.961050677 +0000 UTC m=+1597.934385373" watchObservedRunningTime="2026-11-02 06:44:03.967121118 +0000 UTC m=+1597.940455804" Nov 02 06:44:08.555480 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:08.555428 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:44:09.855741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.855701 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:44:09.861307 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.861285 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:09.943017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.942956 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:09.943202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.943180 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fzw\" (UniqueName: \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:09.943252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.943238 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:09.943378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.943362 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:09.957561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.957534 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8ecb23-serving-cert\"" Nov 02 06:44:09.961499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:09.961473 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:44:10.044667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.044625 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fzw\" (UniqueName: \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.044871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.044697 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.044871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.044817 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.045033 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:10.044943 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-serving-cert: secret "maas-api-e2e-api-a-8ecb23-serving-cert" not found Nov 02 06:44:10.045033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.044944 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.045150 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:10.045030 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls podName:41482f2e-646a-421c-81f8-0c597ccbc412 nodeName:}" failed. No retries permitted until 2026-11-02 06:44:10.545009901 +0000 UTC m=+1604.518344580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls") pod "maas-api-e2e-api-a-8ecb23-6785656864-xbthm" (UID: "41482f2e-646a-421c-81f8-0c597ccbc412") : secret "maas-api-e2e-api-a-8ecb23-serving-cert" not found Nov 02 06:44:10.045537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.045514 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.047328 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.047306 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.101865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.101826 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fzw\" (UniqueName: \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.552864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.552831 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.555274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.555248 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") pod \"maas-api-e2e-api-a-8ecb23-6785656864-xbthm\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:10.776511 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:10.776475 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:11.173534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:11.173508 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:44:11.176488 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:44:11.176463 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41482f2e_646a_421c_81f8_0c597ccbc412.slice/crio-71130319ed593510bf1aec298f325ffff2c7e32bd2a667175792a49db03c2e57 WatchSource:0}: Error finding container 71130319ed593510bf1aec298f325ffff2c7e32bd2a667175792a49db03c2e57: Status 404 returned error can't find the container with id 71130319ed593510bf1aec298f325ffff2c7e32bd2a667175792a49db03c2e57 Nov 02 06:44:11.178310 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:11.178295 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:44:11.809698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:11.809653 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" event={"ID":"41482f2e-646a-421c-81f8-0c597ccbc412","Type":"ContainerStarted","Data":"71130319ed593510bf1aec298f325ffff2c7e32bd2a667175792a49db03c2e57"} Nov 02 06:44:12.818277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:12.818233 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" event={"ID":"41482f2e-646a-421c-81f8-0c597ccbc412","Type":"ContainerStarted","Data":"01c4c7079d113aaeae382b9d03cfbe9219dc0eb18bec0ca43f3aaa31d4f99aed"} Nov 02 06:44:12.818709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:12.818339 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:12.926830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:12.926779 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" podStartSLOduration=2.984046743 podStartE2EDuration="3.926765436s" totalImagesPullingTime="942.718693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:12.925742436 +0000 UTC m=+1606.899077144" watchObservedRunningTime="2026-11-02 06:44:12.926765436 +0000 UTC m=+1606.900100120" Nov 02 06:44:14.764047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:14.764018 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 06:44:14.764603 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:14.764090 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 06:44:18.831972 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:18.831941 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:44:26.784688 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:26.784649 6273 scope.go:122] "RemoveContainer" containerID="f4c93cf56aed6cb150c4f88fb85abdecbebbd97990687b1e2ed31ee119fda52e" Nov 02 06:44:26.795802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:26.795778 6273 scope.go:122] "RemoveContainer" containerID="fc643e3b45c76b840a8704a0b98da76f9848e649b431624990961393a2fc5f82" Nov 02 06:44:28.546676 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.546641 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:44:28.685957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.685917 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:44:28.691831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.691807 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.743893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.743866 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-08bbf9-serving-cert\"" Nov 02 06:44:28.746284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.746258 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:44:28.827043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.826950 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.827211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.827104 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.827211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.827179 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.827353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.827336 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddx87\" (UniqueName: \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.929047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.929009 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.929230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.929065 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.929230 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:28.929137 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-serving-cert: secret "maas-api-e2e-shared-a-w2-08bbf9-serving-cert" not found Nov 02 06:44:28.929230 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:28.929207 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls podName:cf732e9e-6229-4df9-8081-97bd5222f66b nodeName:}" failed. No retries permitted until 2026-11-02 06:44:29.42919188 +0000 UTC m=+1623.402526542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" (UID: "cf732e9e-6229-4df9-8081-97bd5222f66b") : secret "maas-api-e2e-shared-a-w2-08bbf9-serving-cert" not found Nov 02 06:44:28.929382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.929250 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddx87\" (UniqueName: \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.929382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.929358 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.929942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.929921 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.931411 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.931390 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:28.991007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:28.990959 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddx87\" (UniqueName: \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:29.437413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:29.437377 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:29.439840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:29.439814 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:29.606669 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:29.606638 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:30.000190 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:30.000147 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:44:30.006836 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:44:30.006809 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf732e9e_6229_4df9_8081_97bd5222f66b.slice/crio-9ea9cc027aa8a527eb9857baf0e735852b451e0c282b101736dfc274f2efd73f WatchSource:0}: Error finding container 9ea9cc027aa8a527eb9857baf0e735852b451e0c282b101736dfc274f2efd73f: Status 404 returned error can't find the container with id 9ea9cc027aa8a527eb9857baf0e735852b451e0c282b101736dfc274f2efd73f Nov 02 06:44:31.000020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:30.999964 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" event={"ID":"cf732e9e-6229-4df9-8081-97bd5222f66b","Type":"ContainerStarted","Data":"9ea9cc027aa8a527eb9857baf0e735852b451e0c282b101736dfc274f2efd73f"} Nov 02 06:44:32.009561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:32.009521 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" event={"ID":"cf732e9e-6229-4df9-8081-97bd5222f66b","Type":"ContainerStarted","Data":"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8"} Nov 02 06:44:32.009977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:32.009621 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:32.122563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:32.122518 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" podStartSLOduration=3.1566723 podStartE2EDuration="4.122504886s" totalImagesPullingTime="965.832586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:32.119187638 +0000 UTC m=+1626.092522347" watchObservedRunningTime="2026-11-02 06:44:32.122504886 +0000 UTC m=+1626.095839569" Nov 02 06:44:38.023998 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:44:38.023909 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:44:38.555677 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:44:38.555635 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:45:00.158806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.158766 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:00.164004 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.163968 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.183591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.183564 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:00.188682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.188656 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.189195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.189169 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:00.195120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.195078 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.205787 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.205750 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.206068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.206041 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4xs\" (UniqueName: \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.210655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.210634 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:00.215020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.215000 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:00.215119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.215102 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.219299 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.219281 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.231805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.231631 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:00.232177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.232158 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:45:00.273835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.273794 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:00.299152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.299051 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:00.309285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309241 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.309469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309316 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlmf\" (UniqueName: \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.309469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309394 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.309469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309455 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gf4l\" (UniqueName: \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309492 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mphl\" (UniqueName: \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309520 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309542 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309587 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309647 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4xs\" (UniqueName: \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.309704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.309666 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.310215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.310188 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.313665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.313633 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:00.316733 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.316708 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:00.364616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.364578 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4xs\" (UniqueName: \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.410817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410721 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlmf\" (UniqueName: \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.410817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410780 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gf4l\" (UniqueName: \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.410817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410818 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mphl\" (UniqueName: \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.411121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410844 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.411121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410866 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.411121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.410924 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.411121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411034 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.411332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411230 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.411643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411619 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.411643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411632 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.411797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411697 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.411797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.411765 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.465361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.465323 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mphl\" (UniqueName: \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\") pod \"maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.465532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.465373 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\") pod \"maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.465760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.465738 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlmf\" (UniqueName: \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.465833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.465817 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gf4l\" (UniqueName: \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\") pod \"maas-api-key-cleanup-29893365-khghw\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.479784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.479750 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:00.509444 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.509403 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:00.514532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.514493 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:00.531790 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.531755 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:00.538435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.538406 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:00.875655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.875595 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:00.881315 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:00.881285 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685aa46b_4589_4036_87eb_ec19e6249f90.slice/crio-ad44c0a738fd50962d27fc046f8275f6e3ed97e782ae430f5366b1ea9176c9fd WatchSource:0}: Error finding container ad44c0a738fd50962d27fc046f8275f6e3ed97e782ae430f5366b1ea9176c9fd: Status 404 returned error can't find the container with id ad44c0a738fd50962d27fc046f8275f6e3ed97e782ae430f5366b1ea9176c9fd Nov 02 06:45:00.930895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.930811 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:00.934716 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:00.934685 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afd5cb2_935d_4555_8fad_454ba86b7584.slice/crio-4c2d410ac8c92b3c98d66a50b361712ddd661a8b27d3f7f1fea3f7e622f9205a WatchSource:0}: Error finding container 4c2d410ac8c92b3c98d66a50b361712ddd661a8b27d3f7f1fea3f7e622f9205a: Status 404 returned error can't find the container with id 4c2d410ac8c92b3c98d66a50b361712ddd661a8b27d3f7f1fea3f7e622f9205a Nov 02 06:45:00.961548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.961508 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:00.966754 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:00.966707 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38e016d_00a3_4dd9_bca6_c862acbfb5d8.slice/crio-ca6a44ac21fe26b4fef8e7eda639d2f327babc25e91993f157bbe7774a7f0d60 WatchSource:0}: Error finding container ca6a44ac21fe26b4fef8e7eda639d2f327babc25e91993f157bbe7774a7f0d60: Status 404 returned error can't find the container with id ca6a44ac21fe26b4fef8e7eda639d2f327babc25e91993f157bbe7774a7f0d60 Nov 02 06:45:00.984101 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.984064 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:00.990720 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:00.990684 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee278fb_b5fc_424a_b6a4_c6185cef4f36.slice/crio-5c4a5afbd1b5d4351728b803ee12af60dd9a250725724f9709c276c4ecb15359 WatchSource:0}: Error finding container 5c4a5afbd1b5d4351728b803ee12af60dd9a250725724f9709c276c4ecb15359: Status 404 returned error can't find the container with id 5c4a5afbd1b5d4351728b803ee12af60dd9a250725724f9709c276c4ecb15359 Nov 02 06:45:00.991038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:00.990974 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:00.997092 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:00.997064 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b14eb4_5c43_43a7_b481_e2dbc6139a7e.slice/crio-b8c78ffcf3a9f424ff41a2bfebffd0d9f9013017fb44c4284b4f5531274a7f7e WatchSource:0}: Error finding container b8c78ffcf3a9f424ff41a2bfebffd0d9f9013017fb44c4284b4f5531274a7f7e: Status 404 returned error can't find the container with id b8c78ffcf3a9f424ff41a2bfebffd0d9f9013017fb44c4284b4f5531274a7f7e Nov 02 06:45:01.239857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:01.239764 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerStarted","Data":"b8c78ffcf3a9f424ff41a2bfebffd0d9f9013017fb44c4284b4f5531274a7f7e"} Nov 02 06:45:01.241108 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:01.241082 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerStarted","Data":"5c4a5afbd1b5d4351728b803ee12af60dd9a250725724f9709c276c4ecb15359"} Nov 02 06:45:01.242273 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:01.242252 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerStarted","Data":"4c2d410ac8c92b3c98d66a50b361712ddd661a8b27d3f7f1fea3f7e622f9205a"} Nov 02 06:45:01.243408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:01.243389 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" event={"ID":"685aa46b-4589-4036-87eb-ec19e6249f90","Type":"ContainerStarted","Data":"ad44c0a738fd50962d27fc046f8275f6e3ed97e782ae430f5366b1ea9176c9fd"} Nov 02 06:45:01.244550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:01.244529 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" event={"ID":"f38e016d-00a3-4dd9-bca6-c862acbfb5d8","Type":"ContainerStarted","Data":"ca6a44ac21fe26b4fef8e7eda639d2f327babc25e91993f157bbe7774a7f0d60"} Nov 02 06:45:02.253554 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.253518 6273 generic.go:356] "Generic (PLEG): container finished" podID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerID="165a999a64affe74677db2198c61e557325bdb516957738ececc02a984e80a52" exitCode=60 Nov 02 06:45:02.254016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.253599 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerDied","Data":"165a999a64affe74677db2198c61e557325bdb516957738ececc02a984e80a52"} Nov 02 06:45:02.254016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.253930 6273 scope.go:122] "RemoveContainer" containerID="165a999a64affe74677db2198c61e557325bdb516957738ececc02a984e80a52" Nov 02 06:45:02.255395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.255374 6273 generic.go:356] "Generic (PLEG): container finished" podID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerID="7e5f1d3de05e14c211c0b1f53157a24764335efc01106fc522bf6d769e87d986" exitCode=60 Nov 02 06:45:02.255511 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.255459 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerDied","Data":"7e5f1d3de05e14c211c0b1f53157a24764335efc01106fc522bf6d769e87d986"} Nov 02 06:45:02.255760 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.255678 6273 scope.go:122] "RemoveContainer" containerID="7e5f1d3de05e14c211c0b1f53157a24764335efc01106fc522bf6d769e87d986" Nov 02 06:45:02.257144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.257121 6273 generic.go:356] "Generic (PLEG): container finished" podID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerID="04d3b3a0c3aaa48cb466f5fb3023f1fa23803e33f821a2bfedeeda49a7e9c875" exitCode=60 Nov 02 06:45:02.257224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.257146 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerDied","Data":"04d3b3a0c3aaa48cb466f5fb3023f1fa23803e33f821a2bfedeeda49a7e9c875"} Nov 02 06:45:02.257415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.257400 6273 scope.go:122] "RemoveContainer" containerID="04d3b3a0c3aaa48cb466f5fb3023f1fa23803e33f821a2bfedeeda49a7e9c875" Nov 02 06:45:02.258806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.258783 6273 generic.go:356] "Generic (PLEG): container finished" podID="685aa46b-4589-4036-87eb-ec19e6249f90" containerID="47c4f128a1aebc71b0a6a7c56cc701fcfc2a64efb41b51876c772d0f9a0be196" exitCode=60 Nov 02 06:45:02.258919 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.258861 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" event={"ID":"685aa46b-4589-4036-87eb-ec19e6249f90","Type":"ContainerDied","Data":"47c4f128a1aebc71b0a6a7c56cc701fcfc2a64efb41b51876c772d0f9a0be196"} Nov 02 06:45:02.259174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.259070 6273 scope.go:122] "RemoveContainer" containerID="47c4f128a1aebc71b0a6a7c56cc701fcfc2a64efb41b51876c772d0f9a0be196" Nov 02 06:45:02.261487 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.261467 6273 generic.go:356] "Generic (PLEG): container finished" podID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerID="d01558811ef396d73cf1826f2974e8dec36b19b1fe516d18442c3deed276c96c" exitCode=60 Nov 02 06:45:02.261573 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.261493 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" event={"ID":"f38e016d-00a3-4dd9-bca6-c862acbfb5d8","Type":"ContainerDied","Data":"d01558811ef396d73cf1826f2974e8dec36b19b1fe516d18442c3deed276c96c"} Nov 02 06:45:02.261773 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:02.261757 6273 scope.go:122] "RemoveContainer" containerID="d01558811ef396d73cf1826f2974e8dec36b19b1fe516d18442c3deed276c96c" Nov 02 06:45:03.280179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.280124 6273 generic.go:356] "Generic (PLEG): container finished" podID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerID="86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d" exitCode=60 Nov 02 06:45:03.281223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.281191 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" event={"ID":"f38e016d-00a3-4dd9-bca6-c862acbfb5d8","Type":"ContainerDied","Data":"86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d"} Nov 02 06:45:03.281830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.281805 6273 scope.go:122] "RemoveContainer" containerID="d01558811ef396d73cf1826f2974e8dec36b19b1fe516d18442c3deed276c96c" Nov 02 06:45:03.283674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.283512 6273 scope.go:122] "RemoveContainer" containerID="86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d" Nov 02 06:45:03.285822 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:03.284268 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n_odh-ai-gateway-infra(f38e016d-00a3-4dd9-bca6-c862acbfb5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" Nov 02 06:45:03.295900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.295859 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerStarted","Data":"54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd"} Nov 02 06:45:03.298752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.298722 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerStarted","Data":"3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc"} Nov 02 06:45:03.301172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.301141 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerStarted","Data":"ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455"} Nov 02 06:45:03.303151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.303127 6273 generic.go:356] "Generic (PLEG): container finished" podID="685aa46b-4589-4036-87eb-ec19e6249f90" containerID="f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65" exitCode=60 Nov 02 06:45:03.303272 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.303195 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" event={"ID":"685aa46b-4589-4036-87eb-ec19e6249f90","Type":"ContainerDied","Data":"f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65"} Nov 02 06:45:03.303461 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.303445 6273 scope.go:122] "RemoveContainer" containerID="f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65" Nov 02 06:45:03.303731 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:03.303710 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk_odh-ai-gateway-infra(685aa46b-4589-4036-87eb-ec19e6249f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" Nov 02 06:45:03.318408 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.318383 6273 scope.go:122] "RemoveContainer" containerID="47c4f128a1aebc71b0a6a7c56cc701fcfc2a64efb41b51876c772d0f9a0be196" Nov 02 06:45:03.495178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.495124 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" podStartSLOduration=3.495108586 podStartE2EDuration="3.495108586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:03.494611547 +0000 UTC m=+1657.467946231" watchObservedRunningTime="2026-11-02 06:45:03.495108586 +0000 UTC m=+1657.468443270" Nov 02 06:45:03.601856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.601800 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" podStartSLOduration=3.601785375 podStartE2EDuration="3.601785375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:03.60093828 +0000 UTC m=+1657.574272967" watchObservedRunningTime="2026-11-02 06:45:03.601785375 +0000 UTC m=+1657.575120092" Nov 02 06:45:03.710935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:03.710828 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" podStartSLOduration=3.7108127680000003 podStartE2EDuration="3.710812768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:03.708192941 +0000 UTC m=+1657.681527637" watchObservedRunningTime="2026-11-02 06:45:03.710812768 +0000 UTC m=+1657.684147455" Nov 02 06:45:04.312967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.312930 6273 generic.go:356] "Generic (PLEG): container finished" podID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerID="54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd" exitCode=60 Nov 02 06:45:04.313480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.313020 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerDied","Data":"54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd"} Nov 02 06:45:04.313480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.313099 6273 scope.go:122] "RemoveContainer" containerID="165a999a64affe74677db2198c61e557325bdb516957738ececc02a984e80a52" Nov 02 06:45:04.313480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.313428 6273 scope.go:122] "RemoveContainer" containerID="54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd" Nov 02 06:45:04.313870 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:04.313839 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893365-khghw_odh-ai-gateway-infra(15b14eb4-5c43-43a7-b481-e2dbc6139a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" Nov 02 06:45:04.315199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.315176 6273 generic.go:356] "Generic (PLEG): container finished" podID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerID="3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc" exitCode=60 Nov 02 06:45:04.315308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.315262 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerDied","Data":"3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc"} Nov 02 06:45:04.315540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.315522 6273 scope.go:122] "RemoveContainer" containerID="3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc" Nov 02 06:45:04.315870 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:04.315847 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl_odh-ai-gateway-infra(dee278fb-b5fc-424a-b6a4-c6185cef4f36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" Nov 02 06:45:04.317143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.317121 6273 generic.go:356] "Generic (PLEG): container finished" podID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerID="ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455" exitCode=60 Nov 02 06:45:04.317250 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.317202 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerDied","Data":"ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455"} Nov 02 06:45:04.317532 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.317511 6273 scope.go:122] "RemoveContainer" containerID="ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455" Nov 02 06:45:04.317851 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:04.317825 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf_odh-ai-gateway-infra(5afd5cb2-935d-4555-8fad-454ba86b7584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" Nov 02 06:45:04.319279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.319256 6273 scope.go:122] "RemoveContainer" containerID="f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65" Nov 02 06:45:04.319619 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:04.319595 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk_odh-ai-gateway-infra(685aa46b-4589-4036-87eb-ec19e6249f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" Nov 02 06:45:04.320810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.320786 6273 scope.go:122] "RemoveContainer" containerID="86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d" Nov 02 06:45:04.321052 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:04.321026 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n_odh-ai-gateway-infra(f38e016d-00a3-4dd9-bca6-c862acbfb5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" Nov 02 06:45:04.327450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.327431 6273 scope.go:122] "RemoveContainer" containerID="7e5f1d3de05e14c211c0b1f53157a24764335efc01106fc522bf6d769e87d986" Nov 02 06:45:04.341245 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:04.341224 6273 scope.go:122] "RemoveContainer" containerID="04d3b3a0c3aaa48cb466f5fb3023f1fa23803e33f821a2bfedeeda49a7e9c875" Nov 02 06:45:05.330712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:05.330685 6273 scope.go:122] "RemoveContainer" containerID="54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd" Nov 02 06:45:05.331191 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:05.331099 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893365-khghw_odh-ai-gateway-infra(15b14eb4-5c43-43a7-b481-e2dbc6139a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" Nov 02 06:45:05.332280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:05.332255 6273 scope.go:122] "RemoveContainer" containerID="3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc" Nov 02 06:45:05.332599 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:05.332575 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl_odh-ai-gateway-infra(dee278fb-b5fc-424a-b6a4-c6185cef4f36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" Nov 02 06:45:05.333796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:05.333777 6273 scope.go:122] "RemoveContainer" containerID="ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455" Nov 02 06:45:05.334075 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:05.334056 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf_odh-ai-gateway-infra(5afd5cb2-935d-4555-8fad-454ba86b7584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" Nov 02 06:45:07.309976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.309928 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:45:07.320512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.320477 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.353516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.353479 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:45:07.371741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.371708 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:45:07.373556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.373529 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-12bfb5\"" Nov 02 06:45:07.396359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.396306 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-12bfb5-dockercfg-xrtdj\"" Nov 02 06:45:07.434685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.434648 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.434880 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.434845 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zn7\" (UniqueName: \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.461435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.461402 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:45:07.466443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.466416 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.519780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.519738 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:45:07.539732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.537001 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zn7\" (UniqueName: \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.539732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.537269 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.539732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.538236 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.596192 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.596114 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zn7\" (UniqueName: \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\") pod \"payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.637044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.636981 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:07.638653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.638617 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdx42\" (UniqueName: \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.638842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.638768 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.755342 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.755301 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdx42\" (UniqueName: \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.755542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.755465 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.757193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.756668 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:07.809937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:07.809909 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdx42\" (UniqueName: \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\") pod \"payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:08.032332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.032297 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:45:08.038094 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:08.038065 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ced5489_3232_4eb6_8cf9_36977e67c1cd.slice/crio-7362c20b3935295ede45a49d73a385402a7542ff40569bbfe094d3c1e93b7a8f WatchSource:0}: Error finding container 7362c20b3935295ede45a49d73a385402a7542ff40569bbfe094d3c1e93b7a8f: Status 404 returned error can't find the container with id 7362c20b3935295ede45a49d73a385402a7542ff40569bbfe094d3c1e93b7a8f Nov 02 06:45:08.083975 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.083937 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:08.358522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.358444 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" event={"ID":"4ced5489-3232-4eb6-8cf9-36977e67c1cd","Type":"ContainerStarted","Data":"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6"} Nov 02 06:45:08.358522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.358496 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" event={"ID":"4ced5489-3232-4eb6-8cf9-36977e67c1cd","Type":"ContainerStarted","Data":"7362c20b3935295ede45a49d73a385402a7542ff40569bbfe094d3c1e93b7a8f"} Nov 02 06:45:08.358915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.358641 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:08.464647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.464610 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:45:08.468874 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:08.468822 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" podStartSLOduration=1.468808433 podStartE2EDuration="1.468808433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:08.465912085 +0000 UTC m=+1662.439246762" watchObservedRunningTime="2026-11-02 06:45:08.468808433 +0000 UTC m=+1662.442143108" Nov 02 06:45:08.472406 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:08.472378 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8463ed4e_7fbe_4929_9bb8_34a4af15f8ee.slice/crio-b753811720e94ae9be1b601c0dc5ef2d12bc7da633b656b7437d53e276ca05a5 WatchSource:0}: Error finding container b753811720e94ae9be1b601c0dc5ef2d12bc7da633b656b7437d53e276ca05a5: Status 404 returned error can't find the container with id b753811720e94ae9be1b601c0dc5ef2d12bc7da633b656b7437d53e276ca05a5 Nov 02 06:45:08.555581 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:08.555538 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:45:09.368337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:09.368298 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" event={"ID":"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee","Type":"ContainerStarted","Data":"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2"} Nov 02 06:45:09.368784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:09.368366 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" event={"ID":"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee","Type":"ContainerStarted","Data":"b753811720e94ae9be1b601c0dc5ef2d12bc7da633b656b7437d53e276ca05a5"} Nov 02 06:45:09.368784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:09.368670 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:09.478855 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:09.478801 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" podStartSLOduration=2.47878583 podStartE2EDuration="2.47878583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:09.474771021 +0000 UTC m=+1663.448105745" watchObservedRunningTime="2026-11-02 06:45:09.47878583 +0000 UTC m=+1663.452120514" Nov 02 06:45:11.903142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:11.903106 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 06:45:11.907930 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:11.907907 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.003708 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.003672 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-6e2657-serving-cert\"" Nov 02 06:45:12.009367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.009334 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 06:45:12.032832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.032791 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.032966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.032866 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.032966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.032950 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9rd\" (UniqueName: \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.033175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.033149 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.134139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.134106 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.134336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.134162 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.134336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.134217 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9rd\" (UniqueName: \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.134336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.134323 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.134511 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:12.134462 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-serving-cert: secret "maas-api-e2e-shared-b-w6-6e2657-serving-cert" not found Nov 02 06:45:12.134565 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:12.134533 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls podName:03c60f2f-384b-46b2-8c57-38c03f872876 nodeName:}" failed. No retries permitted until 2026-11-02 06:45:12.634513683 +0000 UTC m=+1666.607848365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls") pod "maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" (UID: "03c60f2f-384b-46b2-8c57-38c03f872876") : secret "maas-api-e2e-shared-b-w6-6e2657-serving-cert" not found Nov 02 06:45:12.134818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.134798 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.136712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.136690 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.190423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.190337 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9rd\" (UniqueName: \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.642001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.641893 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.644422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.644396 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:12.826306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:12.826270 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:13.224696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:13.224658 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 06:45:13.230310 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:13.230281 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c60f2f_384b_46b2_8c57_38c03f872876.slice/crio-04ed84418ecc9b22ac2d12f21991412232551a280b8c1f1bd5c66b644065725e WatchSource:0}: Error finding container 04ed84418ecc9b22ac2d12f21991412232551a280b8c1f1bd5c66b644065725e: Status 404 returned error can't find the container with id 04ed84418ecc9b22ac2d12f21991412232551a280b8c1f1bd5c66b644065725e Nov 02 06:45:13.398579 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:13.398546 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" event={"ID":"03c60f2f-384b-46b2-8c57-38c03f872876","Type":"ContainerStarted","Data":"04ed84418ecc9b22ac2d12f21991412232551a280b8c1f1bd5c66b644065725e"} Nov 02 06:45:14.172164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:14.172137 6273 scope.go:122] "RemoveContainer" containerID="3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc" Nov 02 06:45:14.407932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:14.407896 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" event={"ID":"03c60f2f-384b-46b2-8c57-38c03f872876","Type":"ContainerStarted","Data":"a7003fe3d2beb1a1cff783aad45ac129214cd078c84d6d2c8e4feaa36a833476"} Nov 02 06:45:14.408414 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:14.408031 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:14.516898 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:14.516837 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" podStartSLOduration=2.617576833 podStartE2EDuration="3.516783847s" totalImagesPullingTime="899.207014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:14.516499535 +0000 UTC m=+1668.489834220" watchObservedRunningTime="2026-11-02 06:45:14.516783847 +0000 UTC m=+1668.490118528" Nov 02 06:45:15.172726 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:15.172701 6273 scope.go:122] "RemoveContainer" containerID="ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455" Nov 02 06:45:15.417537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:15.417501 6273 generic.go:356] "Generic (PLEG): container finished" podID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerID="3c163702d0a35e0f48c1f30c854b9381d88b17e90764687a1f49c5b15da953aa" exitCode=60 Nov 02 06:45:15.418017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:15.417572 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerDied","Data":"3c163702d0a35e0f48c1f30c854b9381d88b17e90764687a1f49c5b15da953aa"} Nov 02 06:45:15.418017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:15.417629 6273 scope.go:122] "RemoveContainer" containerID="3574c8011c1be9ee9335f80360602a3d98e1af9a943816c99e4f524839bd5efc" Nov 02 06:45:15.418157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:15.418108 6273 scope.go:122] "RemoveContainer" containerID="3c163702d0a35e0f48c1f30c854b9381d88b17e90764687a1f49c5b15da953aa" Nov 02 06:45:15.418537 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:15.418515 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl_odh-ai-gateway-infra(dee278fb-b5fc-424a-b6a4-c6185cef4f36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" Nov 02 06:45:16.177513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.177479 6273 scope.go:122] "RemoveContainer" containerID="86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d" Nov 02 06:45:16.177783 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.177763 6273 scope.go:122] "RemoveContainer" containerID="54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd" Nov 02 06:45:16.177904 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.177827 6273 scope.go:122] "RemoveContainer" containerID="f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65" Nov 02 06:45:16.429277 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.429242 6273 generic.go:356] "Generic (PLEG): container finished" podID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerID="c2961c3c579204b6753b0d113bd392d2dc2fe7dd61d472b9016971a788a47cc7" exitCode=60 Nov 02 06:45:16.429757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.429312 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerDied","Data":"c2961c3c579204b6753b0d113bd392d2dc2fe7dd61d472b9016971a788a47cc7"} Nov 02 06:45:16.429757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.429382 6273 scope.go:122] "RemoveContainer" containerID="ea0b8ad6a3445aa9bdbe65f70b5296bd3284a9da0f1f08737599f89dcb873455" Nov 02 06:45:16.429870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.429840 6273 scope.go:122] "RemoveContainer" containerID="c2961c3c579204b6753b0d113bd392d2dc2fe7dd61d472b9016971a788a47cc7" Nov 02 06:45:16.430269 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:16.430249 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf_odh-ai-gateway-infra(5afd5cb2-935d-4555-8fad-454ba86b7584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" Nov 02 06:45:16.539058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.539026 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:16.772182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.772156 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:16.910261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.910204 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\") pod \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " Nov 02 06:45:16.910470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.910354 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\") pod \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\" (UID: \"dee278fb-b5fc-424a-b6a4-c6185cef4f36\") " Nov 02 06:45:16.910647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.910621 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca" pod "dee278fb-b5fc-424a-b6a4-c6185cef4f36" (UID: "dee278fb-b5fc-424a-b6a4-c6185cef4f36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:16.911204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.911180 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee278fb-b5fc-424a-b6a4-c6185cef4f36-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:16.916186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:16.916154 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh" pod "dee278fb-b5fc-424a-b6a4-c6185cef4f36" (UID: "dee278fb-b5fc-424a-b6a4-c6185cef4f36"). InnerVolumeSpecName "kube-api-access-bljdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:17.013047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.012952 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/dee278fb-b5fc-424a-b6a4-c6185cef4f36-kube-api-access-bljdh\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:17.439229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.439196 6273 generic.go:356] "Generic (PLEG): container finished" podID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerID="e50b68ac5865581a8bd577d136e965d8fc22ddffbedcfb147c36f5032c46bcd5" exitCode=60 Nov 02 06:45:17.439684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.439267 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerDied","Data":"e50b68ac5865581a8bd577d136e965d8fc22ddffbedcfb147c36f5032c46bcd5"} Nov 02 06:45:17.439684 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.439368 6273 scope.go:122] "RemoveContainer" containerID="54b92d023f381e8506938b6348710867db31b8332e1d73893bab14fc6bdeffcd" Nov 02 06:45:17.439839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.439820 6273 scope.go:122] "RemoveContainer" containerID="e50b68ac5865581a8bd577d136e965d8fc22ddffbedcfb147c36f5032c46bcd5" Nov 02 06:45:17.440203 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:17.440181 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893365-khghw_odh-ai-gateway-infra(15b14eb4-5c43-43a7-b481-e2dbc6139a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" Nov 02 06:45:17.441536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.441512 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" event={"ID":"dee278fb-b5fc-424a-b6a4-c6185cef4f36","Type":"ContainerDied","Data":"5c4a5afbd1b5d4351728b803ee12af60dd9a250725724f9709c276c4ecb15359"} Nov 02 06:45:17.441536 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.441531 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl" Nov 02 06:45:17.444984 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.444963 6273 generic.go:356] "Generic (PLEG): container finished" podID="685aa46b-4589-4036-87eb-ec19e6249f90" containerID="a9218e816370e3c70a783b3fac66ebae1f2435ac1c9766fae2941de6e5387bf8" exitCode=60 Nov 02 06:45:17.445090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.445022 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" event={"ID":"685aa46b-4589-4036-87eb-ec19e6249f90","Type":"ContainerDied","Data":"a9218e816370e3c70a783b3fac66ebae1f2435ac1c9766fae2941de6e5387bf8"} Nov 02 06:45:17.445466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.445450 6273 scope.go:122] "RemoveContainer" containerID="a9218e816370e3c70a783b3fac66ebae1f2435ac1c9766fae2941de6e5387bf8" Nov 02 06:45:17.445803 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:17.445782 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk_odh-ai-gateway-infra(685aa46b-4589-4036-87eb-ec19e6249f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" Nov 02 06:45:17.447126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.447108 6273 generic.go:356] "Generic (PLEG): container finished" podID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerID="819f485d6c1a38fd8fd3b77fbce9fb5a2f783ce559f839af459c66a51c4c0ea1" exitCode=60 Nov 02 06:45:17.447207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.447184 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" event={"ID":"f38e016d-00a3-4dd9-bca6-c862acbfb5d8","Type":"ContainerDied","Data":"819f485d6c1a38fd8fd3b77fbce9fb5a2f783ce559f839af459c66a51c4c0ea1"} Nov 02 06:45:17.447543 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.447525 6273 scope.go:122] "RemoveContainer" containerID="819f485d6c1a38fd8fd3b77fbce9fb5a2f783ce559f839af459c66a51c4c0ea1" Nov 02 06:45:17.447817 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:17.447797 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n_odh-ai-gateway-infra(f38e016d-00a3-4dd9-bca6-c862acbfb5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" Nov 02 06:45:17.455770 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.455750 6273 scope.go:122] "RemoveContainer" containerID="3c163702d0a35e0f48c1f30c854b9381d88b17e90764687a1f49c5b15da953aa" Nov 02 06:45:17.497152 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.497128 6273 scope.go:122] "RemoveContainer" containerID="f57938b23275efcf57a904ebfb857466499c7dabb03102aabc6b4e398d79ba65" Nov 02 06:45:17.510831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.510811 6273 scope.go:122] "RemoveContainer" containerID="86608aee396ce73cf298f06511dc1661508ee2e3570e37633284cf6c34db228d" Nov 02 06:45:17.554154 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.554117 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:17.730767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.730732 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:17.825121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.825037 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\") pod \"5afd5cb2-935d-4555-8fad-454ba86b7584\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " Nov 02 06:45:17.825267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.825230 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\" (UniqueName: \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\") pod \"5afd5cb2-935d-4555-8fad-454ba86b7584\" (UID: \"5afd5cb2-935d-4555-8fad-454ba86b7584\") " Nov 02 06:45:17.825508 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.825485 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca" pod "5afd5cb2-935d-4555-8fad-454ba86b7584" (UID: "5afd5cb2-935d-4555-8fad-454ba86b7584"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:17.825850 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.825834 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afd5cb2-935d-4555-8fad-454ba86b7584-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:17.827229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.827207 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl" pod "5afd5cb2-935d-4555-8fad-454ba86b7584" (UID: "5afd5cb2-935d-4555-8fad-454ba86b7584"). InnerVolumeSpecName "kube-api-access-2mphl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:17.869232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.869196 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:17.872368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.872292 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e5502564-29893365-kxbpl"] Nov 02 06:45:17.927673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:17.927634 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mphl\" (UniqueName: \"kubernetes.io/projected/5afd5cb2-935d-4555-8fad-454ba86b7584-kube-api-access-2mphl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.201294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.201253 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" path="/var/lib/kubelet/pods/dee278fb-b5fc-424a-b6a4-c6185cef4f36/volumes" Nov 02 06:45:18.460042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.459935 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" Nov 02 06:45:18.460042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.459941 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf" event={"ID":"5afd5cb2-935d-4555-8fad-454ba86b7584","Type":"ContainerDied","Data":"4c2d410ac8c92b3c98d66a50b361712ddd661a8b27d3f7f1fea3f7e622f9205a"} Nov 02 06:45:18.460569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.460075 6273 scope.go:122] "RemoveContainer" containerID="c2961c3c579204b6753b0d113bd392d2dc2fe7dd61d472b9016971a788a47cc7" Nov 02 06:45:18.569338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.569306 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:18.620138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.620103 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:18.630600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.630572 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3bf02ebf-29893365-zjnlf"] Nov 02 06:45:18.669037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.667137 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:18.768445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.768067 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:18.783413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.783381 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:18.842052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.841205 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\" (UniqueName: \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\") pod \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " Nov 02 06:45:18.842052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.841330 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\") pod \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\" (UID: \"15b14eb4-5c43-43a7-b481-e2dbc6139a7e\") " Nov 02 06:45:18.842423 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.842401 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca" pod "15b14eb4-5c43-43a7-b481-e2dbc6139a7e" (UID: "15b14eb4-5c43-43a7-b481-e2dbc6139a7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:18.844425 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.844401 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l" pod "15b14eb4-5c43-43a7-b481-e2dbc6139a7e" (UID: "15b14eb4-5c43-43a7-b481-e2dbc6139a7e"). InnerVolumeSpecName "kube-api-access-8gf4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:18.856323 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.856295 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:18.934944 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.934919 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:18.943206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.943180 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\") pod \"685aa46b-4589-4036-87eb-ec19e6249f90\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " Nov 02 06:45:18.943434 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.943417 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\" (UniqueName: \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\") pod \"685aa46b-4589-4036-87eb-ec19e6249f90\" (UID: \"685aa46b-4589-4036-87eb-ec19e6249f90\") " Nov 02 06:45:18.943512 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.943490 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca" pod "685aa46b-4589-4036-87eb-ec19e6249f90" (UID: "685aa46b-4589-4036-87eb-ec19e6249f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:18.944016 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.943980 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/685aa46b-4589-4036-87eb-ec19e6249f90-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.944088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.944020 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gf4l\" (UniqueName: \"kubernetes.io/projected/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-kube-api-access-8gf4l\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.944088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.944038 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b14eb4-5c43-43a7-b481-e2dbc6139a7e-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.945382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:18.945363 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs" pod "685aa46b-4589-4036-87eb-ec19e6249f90" (UID: "685aa46b-4589-4036-87eb-ec19e6249f90"). InnerVolumeSpecName "kube-api-access-4h4xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.045310 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.045209 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\") pod \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " Nov 02 06:45:19.045465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.045364 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\" (UniqueName: \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\") pod \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\" (UID: \"f38e016d-00a3-4dd9-bca6-c862acbfb5d8\") " Nov 02 06:45:19.045590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.045564 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca" pod "f38e016d-00a3-4dd9-bca6-c862acbfb5d8" (UID: "f38e016d-00a3-4dd9-bca6-c862acbfb5d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.045915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.045898 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4xs\" (UniqueName: \"kubernetes.io/projected/685aa46b-4589-4036-87eb-ec19e6249f90-kube-api-access-4h4xs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.046012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.045918 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.047585 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.047563 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf" pod "f38e016d-00a3-4dd9-bca6-c862acbfb5d8" (UID: "f38e016d-00a3-4dd9-bca6-c862acbfb5d8"). InnerVolumeSpecName "kube-api-access-wzlmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.147605 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.147565 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlmf\" (UniqueName: \"kubernetes.io/projected/f38e016d-00a3-4dd9-bca6-c862acbfb5d8-kube-api-access-wzlmf\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.371763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.371683 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:45:19.473571 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.473536 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" event={"ID":"15b14eb4-5c43-43a7-b481-e2dbc6139a7e","Type":"ContainerDied","Data":"b8c78ffcf3a9f424ff41a2bfebffd0d9f9013017fb44c4284b4f5531274a7f7e"} Nov 02 06:45:19.474070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.473606 6273 scope.go:122] "RemoveContainer" containerID="e50b68ac5865581a8bd577d136e965d8fc22ddffbedcfb147c36f5032c46bcd5" Nov 02 06:45:19.474070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.473702 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw" Nov 02 06:45:19.486082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.486054 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" Nov 02 06:45:19.486213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.486128 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk" event={"ID":"685aa46b-4589-4036-87eb-ec19e6249f90","Type":"ContainerDied","Data":"ad44c0a738fd50962d27fc046f8275f6e3ed97e782ae430f5366b1ea9176c9fd"} Nov 02 06:45:19.506922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.501053 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" event={"ID":"f38e016d-00a3-4dd9-bca6-c862acbfb5d8","Type":"ContainerDied","Data":"ca6a44ac21fe26b4fef8e7eda639d2f327babc25e91993f157bbe7774a7f0d60"} Nov 02 06:45:19.506922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.501311 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n" Nov 02 06:45:19.511797 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.511771 6273 scope.go:122] "RemoveContainer" containerID="a9218e816370e3c70a783b3fac66ebae1f2435ac1c9766fae2941de6e5387bf8" Nov 02 06:45:19.529606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.529579 6273 scope.go:122] "RemoveContainer" containerID="819f485d6c1a38fd8fd3b77fbce9fb5a2f783ce559f839af459c66a51c4c0ea1" Nov 02 06:45:19.652913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.652845 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:19.656428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.656401 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893365-bg5nk"] Nov 02 06:45:19.800370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.800335 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:19.803558 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.803529 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893365-khghw"] Nov 02 06:45:19.910061 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.909939 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:19.913870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:19.913840 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-08bbf9-29893365-2np7n"] Nov 02 06:45:20.182504 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.182388 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" path="/var/lib/kubelet/pods/15b14eb4-5c43-43a7-b481-e2dbc6139a7e/volumes" Nov 02 06:45:20.182969 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.182947 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" path="/var/lib/kubelet/pods/5afd5cb2-935d-4555-8fad-454ba86b7584/volumes" Nov 02 06:45:20.183428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.183413 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" path="/var/lib/kubelet/pods/685aa46b-4589-4036-87eb-ec19e6249f90/volumes" Nov 02 06:45:20.184130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.184115 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" path="/var/lib/kubelet/pods/f38e016d-00a3-4dd9-bca6-c862acbfb5d8/volumes" Nov 02 06:45:20.379600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.379571 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:45:20.423580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:20.423550 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 06:45:34.952687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:34.952650 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:45:38.555023 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:45:38.554971 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:45:39.953639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:39.953582 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" containerID="cri-o://cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3" gracePeriod=25 Nov 02 06:45:42.553839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:42.553799 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:45:48.878944 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.878909 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:45:48.880019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.879961 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.880019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880003 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.880019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880022 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880031 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880049 6273 cpu_manager.go:414] "containerMap: removing container" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880057 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880083 6273 cpu_manager.go:414] "containerMap: removing container" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880091 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880106 6273 cpu_manager.go:414] "containerMap: removing container" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880113 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880126 6273 cpu_manager.go:414] "containerMap: removing container" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880133 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880142 6273 cpu_manager.go:414] "containerMap: removing container" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880149 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880165 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880172 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880186 6273 cpu_manager.go:414] "containerMap: removing container" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880192 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880204 6273 cpu_manager.go:414] "containerMap: removing container" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880211 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880222 6273 cpu_manager.go:414] "containerMap: removing container" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880229 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880240 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.880248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880247 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880428 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880439 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880449 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880462 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880473 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880483 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880493 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880505 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b14eb4-5c43-43a7-b481-e2dbc6139a7e" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880514 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880523 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee278fb-b5fc-424a-b6a4-c6185cef4f36" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880531 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:45:48.881038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.880543 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:45:48.886261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.886218 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:48.929738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.929701 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:48.929906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.929833 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p86x\" (UniqueName: \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:48.938117 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.938083 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:45:48.980610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.980572 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-37b53e-dockercfg-ghlvn\"" Nov 02 06:45:48.981054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:48.980941 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-37b53e\"" Nov 02 06:45:49.031100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.031061 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.031320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.031228 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p86x\" (UniqueName: \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.031919 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.031889 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.090365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.090326 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p86x\" (UniqueName: \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\") pod \"payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.206213 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.206174 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.613261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.613173 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:45:49.618668 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:45:49.618636 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a1ec1a_144d_4116_9b04_afcb89c3ed00.slice/crio-c573a478f6c29b237713dc9e7e99f700c29ada2ebdd0bbfc0744f40dff8f3bec WatchSource:0}: Error finding container c573a478f6c29b237713dc9e7e99f700c29ada2ebdd0bbfc0744f40dff8f3bec: Status 404 returned error can't find the container with id c573a478f6c29b237713dc9e7e99f700c29ada2ebdd0bbfc0744f40dff8f3bec Nov 02 06:45:49.757399 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.757364 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" event={"ID":"d6a1ec1a-144d-4116-9b04-afcb89c3ed00","Type":"ContainerStarted","Data":"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b"} Nov 02 06:45:49.757581 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.757415 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" event={"ID":"d6a1ec1a-144d-4116-9b04-afcb89c3ed00","Type":"ContainerStarted","Data":"c573a478f6c29b237713dc9e7e99f700c29ada2ebdd0bbfc0744f40dff8f3bec"} Nov 02 06:45:49.757581 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.757503 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:45:49.816450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:49.816413 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:45:52.553717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:45:52.553679 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:46:00.344720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.344686 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:46:00.434521 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.434481 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\") pod \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " Nov 02 06:46:00.434727 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.434541 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\" (UniqueName: \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\") pod \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\" (UID: \"1606d8c1-f079-4b00-96a4-dcf91c28e2f8\") " Nov 02 06:46:00.434916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.434888 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume" pod "1606d8c1-f079-4b00-96a4-dcf91c28e2f8" (UID: "1606d8c1-f079-4b00-96a4-dcf91c28e2f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:00.435366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.435346 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:00.436801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.436769 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w" pod "1606d8c1-f079-4b00-96a4-dcf91c28e2f8" (UID: "1606d8c1-f079-4b00-96a4-dcf91c28e2f8"). InnerVolumeSpecName "kube-api-access-q4p4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:00.453459 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.453397 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" podStartSLOduration=12.453378869 podStartE2EDuration="12.453378869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:49.884368513 +0000 UTC m=+1703.857703197" watchObservedRunningTime="2026-11-02 06:46:00.453378869 +0000 UTC m=+1714.426713558" Nov 02 06:46:00.537178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.537141 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4p4w\" (UniqueName: \"kubernetes.io/projected/1606d8c1-f079-4b00-96a4-dcf91c28e2f8-kube-api-access-q4p4w\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:00.768814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.768777 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:46:00.835096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.835060 6273 generic.go:356] "Generic (PLEG): container finished" podID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerID="cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3" exitCode=0 Nov 02 06:46:00.835281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.835119 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" Nov 02 06:46:00.835281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.835137 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" event={"ID":"1606d8c1-f079-4b00-96a4-dcf91c28e2f8","Type":"ContainerDied","Data":"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3"} Nov 02 06:46:00.835281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.835201 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n" event={"ID":"1606d8c1-f079-4b00-96a4-dcf91c28e2f8","Type":"ContainerDied","Data":"14d2b64abf76741261842304098d628e31a7763c2ae6cfc1c179b3dc7bb31e19"} Nov 02 06:46:00.835281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.835233 6273 scope.go:122] "RemoveContainer" containerID="cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3" Nov 02 06:46:00.851091 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.851066 6273 scope.go:122] "RemoveContainer" containerID="cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3" Nov 02 06:46:00.851404 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:00.851381 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3\": container with ID starting with cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3 not found: ID does not exist" containerID="cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3" Nov 02 06:46:00.851475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.851417 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3"} err="failed to get container status \"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3\": rpc error: code = NotFound desc = could not find container \"cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3\": container with ID starting with cc319eca120345855f191a8d3ef49d28125be46e85bdf8c73c31fe47fbc0e5f3 not found: ID does not exist" Nov 02 06:46:00.990979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:00.990942 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:46:01.003393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:01.003360 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-2sp4n"] Nov 02 06:46:02.181656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:02.181619 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" path="/var/lib/kubelet/pods/1606d8c1-f079-4b00-96a4-dcf91c28e2f8/volumes" Nov 02 06:46:05.067705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:05.067618 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:46:05.068210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:05.067856 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" containerID="cri-o://eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b" gracePeriod=30 Nov 02 06:46:05.377577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:05.377490 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:46:05.377733 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:05.377690 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" containerID="cri-o://98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d" gracePeriod=30 Nov 02 06:46:06.478280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:06.478244 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:46:06.478757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:06.478518 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" containerID="cri-o://24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7" gracePeriod=30 Nov 02 06:46:08.554447 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:08.554405 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:46:08.899372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.899297 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:46:08.900101 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900084 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:46:08.900158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900103 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:46:08.900158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900122 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" Nov 02 06:46:08.900158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900127 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" Nov 02 06:46:08.900158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900147 6273 cpu_manager.go:414] "containerMap: removing container" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:46:08.900158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900152 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:46:08.900314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900163 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:46:08.900314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900168 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:46:08.900314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900281 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afd5cb2-935d-4555-8fad-454ba86b7584" containerName="cleanup" Nov 02 06:46:08.900314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900295 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1606d8c1-f079-4b00-96a4-dcf91c28e2f8" containerName="payload-pre-processing" Nov 02 06:46:08.900314 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900307 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="685aa46b-4589-4036-87eb-ec19e6249f90" containerName="cleanup" Nov 02 06:46:08.900460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.900320 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38e016d-00a3-4dd9-bca6-c862acbfb5d8" containerName="cleanup" Nov 02 06:46:08.905444 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.905426 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:08.958641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.958610 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:46:08.962676 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:08.962649 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6d13c3\"" Nov 02 06:46:09.001308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.001278 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6d13c3-dockercfg-xfltd\"" Nov 02 06:46:09.060683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.060648 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcmh\" (UniqueName: \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.060907 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.060864 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.165042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.163097 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcmh\" (UniqueName: \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.165042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.163355 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.165042 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.164472 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.218798 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.218766 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcmh\" (UniqueName: \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\") pod \"payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.219668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.219645 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:09.609085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.609047 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:46:09.617200 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:46:09.617174 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992a7513_01ed_4312_8920_138ee72d3e38.slice/crio-bf94151a4bd26590ecb277c7c88ae72def12dfca3bf4eca421f5e204bbbffc67 WatchSource:0}: Error finding container bf94151a4bd26590ecb277c7c88ae72def12dfca3bf4eca421f5e204bbbffc67: Status 404 returned error can't find the container with id bf94151a4bd26590ecb277c7c88ae72def12dfca3bf4eca421f5e204bbbffc67 Nov 02 06:46:09.906363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.906330 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" event={"ID":"992a7513-01ed-4312-8920-138ee72d3e38","Type":"ContainerStarted","Data":"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493"} Nov 02 06:46:09.906522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.906384 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" event={"ID":"992a7513-01ed-4312-8920-138ee72d3e38","Type":"ContainerStarted","Data":"bf94151a4bd26590ecb277c7c88ae72def12dfca3bf4eca421f5e204bbbffc67"} Nov 02 06:46:09.906522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:09.906517 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:10.016584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.016477 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" podStartSLOduration=2.01646344 podStartE2EDuration="2.01646344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:10.014261091 +0000 UTC m=+1723.987595799" watchObservedRunningTime="2026-11-02 06:46:10.01646344 +0000 UTC m=+1723.989798124" Nov 02 06:46:10.362621 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.362588 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:46:10.482884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.482854 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.482973 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483042 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483133 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483330 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483174 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483394 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483376 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483484 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483444 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483529 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483620 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\") pod \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\" (UID: \"24049b0f-974b-40b9-a7dd-5cfc2d137ebf\") " Nov 02 06:46:10.483694 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483617 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.483971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.483945 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:10.484278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484253 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.484372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484345 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.484531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484511 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.484744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484538 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.484744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484555 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.484744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484569 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.484744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.484642 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.486636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.486608 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.486736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.486662 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "kube-api-access-nms4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.486810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.486786 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:10.487111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.487093 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token" pod "24049b0f-974b-40b9-a7dd-5cfc2d137ebf" (UID: "24049b0f-974b-40b9-a7dd-5cfc2d137ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.513331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.513291 6273 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 06:46:10.513452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.513349 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 06:46:10.589404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.589244 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.589404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.589278 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.589404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.589292 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.589404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.589305 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.589404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.589320 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nms4p\" (UniqueName: \"kubernetes.io/projected/24049b0f-974b-40b9-a7dd-5cfc2d137ebf-kube-api-access-nms4p\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.639794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.639768 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:46:10.791954 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.791924 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792006 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792119 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792184 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792296 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792329 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:10.792378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792347 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.792478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792455 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.792528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792484 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792590 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792566 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792601 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792698 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\") pod \"a8051d6e-2e6c-4763-a660-bdc672eee185\" (UID: \"a8051d6e-2e6c-4763-a660-bdc672eee185\") " Nov 02 06:46:10.792841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792819 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.792931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.792904 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.793491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.793459 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.793491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.793488 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.793673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.793505 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.793673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.793519 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8051d6e-2e6c-4763-a660-bdc672eee185-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.793673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.793533 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.794749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.794721 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:10.794959 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.794935 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.795102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.795082 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "kube-api-access-blrvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.795168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.795142 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token" pod "a8051d6e-2e6c-4763-a660-bdc672eee185" (UID: "a8051d6e-2e6c-4763-a660-bdc672eee185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.895321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.895250 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.895321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.895278 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.895321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.895290 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrvx\" (UniqueName: \"kubernetes.io/projected/a8051d6e-2e6c-4763-a660-bdc672eee185-kube-api-access-blrvx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.895321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.895300 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8051d6e-2e6c-4763-a660-bdc672eee185-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.915186 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.915160 6273 generic.go:356] "Generic (PLEG): container finished" podID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerID="eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b" exitCode=0 Nov 02 06:46:10.915311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.915210 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" Nov 02 06:46:10.915311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.915244 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" event={"ID":"24049b0f-974b-40b9-a7dd-5cfc2d137ebf","Type":"ContainerDied","Data":"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b"} Nov 02 06:46:10.915311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.915306 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn" event={"ID":"24049b0f-974b-40b9-a7dd-5cfc2d137ebf","Type":"ContainerDied","Data":"0399234d7c088df84aa074caf838f6c8aa4ecb8a09497cf330ed9c51d1793daf"} Nov 02 06:46:10.915452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.915339 6273 scope.go:122] "RemoveContainer" containerID="eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b" Nov 02 06:46:10.916699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.916676 6273 generic.go:356] "Generic (PLEG): container finished" podID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerID="98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d" exitCode=0 Nov 02 06:46:10.916771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.916726 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" Nov 02 06:46:10.916771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.916758 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" event={"ID":"a8051d6e-2e6c-4763-a660-bdc672eee185","Type":"ContainerDied","Data":"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d"} Nov 02 06:46:10.916864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.916817 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz" event={"ID":"a8051d6e-2e6c-4763-a660-bdc672eee185","Type":"ContainerDied","Data":"dfd0d41673c7ea803b04bfebf2f83990ceec2c81c855fbccfa30685c22fb6fcd"} Nov 02 06:46:10.926313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.926266 6273 scope.go:122] "RemoveContainer" containerID="eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b" Nov 02 06:46:10.926574 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:10.926550 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b\": container with ID starting with eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b not found: ID does not exist" containerID="eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b" Nov 02 06:46:10.926646 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.926586 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b"} err="failed to get container status \"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b\": rpc error: code = NotFound desc = could not find container \"eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b\": container with ID starting with eebb1c6476c5829eb1e80377d1d83bb1d496dfe168fd600febfa48b6d864eb8b not found: ID does not exist" Nov 02 06:46:10.926646 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.926610 6273 scope.go:122] "RemoveContainer" containerID="98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d" Nov 02 06:46:10.935938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.935916 6273 scope.go:122] "RemoveContainer" containerID="98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d" Nov 02 06:46:10.936246 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:10.936223 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d\": container with ID starting with 98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d not found: ID does not exist" containerID="98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d" Nov 02 06:46:10.936330 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:10.936253 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d"} err="failed to get container status \"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d\": rpc error: code = NotFound desc = could not find container \"98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d\": container with ID starting with 98975354fd61b382d9d69212995e8d52b0f9e3b9e1da677a4a8a5aeb89f7ea1d not found: ID does not exist" Nov 02 06:46:11.079498 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.079472 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:46:11.086033 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.086002 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-k4mfn"] Nov 02 06:46:11.190251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.190189 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:46:11.195202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.195164 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-wsnxz"] Nov 02 06:46:11.745032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.744969 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:46:11.924627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.924578 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.924846 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.924660 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.924846 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.924692 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.924846 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.924769 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.925097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.924956 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.925097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.925019 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.925097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.925056 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.925274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.925139 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.925274 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.925179 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\") pod \"4854653d-ecd4-49dc-8924-67b9c1f96241\" (UID: \"4854653d-ecd4-49dc-8924-67b9c1f96241\") " Nov 02 06:46:11.939051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.936313 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.939051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.937932 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.939051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.937949 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.939051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.938288 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.939051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.938353 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:11.941206 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.941162 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.943631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.943592 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:11.946437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.946411 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "kube-api-access-n7nbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.962106 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.962076 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy" pod "4854653d-ecd4-49dc-8924-67b9c1f96241" (UID: "4854653d-ecd4-49dc-8924-67b9c1f96241"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.968706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.968673 6273 generic.go:356] "Generic (PLEG): container finished" podID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerID="24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7" exitCode=0 Nov 02 06:46:11.968853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.968750 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" Nov 02 06:46:11.968853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.968755 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" event={"ID":"4854653d-ecd4-49dc-8924-67b9c1f96241","Type":"ContainerDied","Data":"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7"} Nov 02 06:46:11.968973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.968854 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq" event={"ID":"4854653d-ecd4-49dc-8924-67b9c1f96241","Type":"ContainerDied","Data":"2594b0f6557f79b55cc4fb99cc4c39c4e67a62116e4c385748455959e8eb1bce"} Nov 02 06:46:11.968973 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.968888 6273 scope.go:122] "RemoveContainer" containerID="24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7" Nov 02 06:46:11.985236 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.985217 6273 scope.go:122] "RemoveContainer" containerID="24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7" Nov 02 06:46:11.985521 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:11.985502 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7\": container with ID starting with 24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7 not found: ID does not exist" containerID="24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7" Nov 02 06:46:11.985593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:11.985528 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7"} err="failed to get container status \"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7\": rpc error: code = NotFound desc = could not find container \"24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7\": container with ID starting with 24039a281dea1912df035e329105f8582491d6f0ae1937ace2db17ff06de99a7 not found: ID does not exist" Nov 02 06:46:12.027678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027623 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7nbd\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-kube-api-access-n7nbd\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027647 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027659 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027669 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027678 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027687 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027716 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4854653d-ecd4-49dc-8924-67b9c1f96241-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027726 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.027905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.027735 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4854653d-ecd4-49dc-8924-67b9c1f96241-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.125468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.125188 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:46:12.128683 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.128648 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-j4bbq"] Nov 02 06:46:12.181841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.181807 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" path="/var/lib/kubelet/pods/24049b0f-974b-40b9-a7dd-5cfc2d137ebf/volumes" Nov 02 06:46:12.182342 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.182325 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" path="/var/lib/kubelet/pods/4854653d-ecd4-49dc-8924-67b9c1f96241/volumes" Nov 02 06:46:12.182786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:12.182772 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" path="/var/lib/kubelet/pods/a8051d6e-2e6c-4763-a660-bdc672eee185/volumes" Nov 02 06:46:20.921240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:20.921210 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:46:22.678208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678178 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:46:22.678856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678841 6273 cpu_manager.go:414] "containerMap: removing container" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" Nov 02 06:46:22.678902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678858 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" Nov 02 06:46:22.678902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678878 6273 cpu_manager.go:414] "containerMap: removing container" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" Nov 02 06:46:22.678902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678883 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" Nov 02 06:46:22.678902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678894 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" Nov 02 06:46:22.678902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.678900 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" Nov 02 06:46:22.679121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.679106 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="24049b0f-974b-40b9-a7dd-5cfc2d137ebf" containerName="istio-proxy" Nov 02 06:46:22.679168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.679132 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="4854653d-ecd4-49dc-8924-67b9c1f96241" containerName="istio-proxy" Nov 02 06:46:22.679168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.679146 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8051d6e-2e6c-4763-a660-bdc672eee185" containerName="istio-proxy" Nov 02 06:46:22.683171 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.683150 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.767447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.767413 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.767696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.767672 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.767828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.767805 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r68h\" (UniqueName: \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.767939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.767926 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.780826 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.780799 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-12bfb5-serving-cert\"" Nov 02 06:46:22.784002 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.783965 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:46:22.869212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.869189 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.869327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.869259 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.869398 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.869383 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.869460 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.869444 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r68h\" (UniqueName: \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.869546 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:22.869528 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-serving-cert: secret "maas-api-e2e-inf-a-12bfb5-serving-cert" not found Nov 02 06:46:22.869604 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:22.869590 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls podName:2f98bd5a-55a2-46ee-86b6-d13ea60af7fb nodeName:}" failed. No retries permitted until 2026-11-02 06:46:23.369573884 +0000 UTC m=+1737.342908545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls") pod "maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" (UID: "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb") : secret "maas-api-e2e-inf-a-12bfb5-serving-cert" not found Nov 02 06:46:22.869835 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.869813 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.871589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.871572 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:22.924559 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:22.924525 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r68h\" (UniqueName: \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:23.376653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:23.376618 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:23.379418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:23.379391 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:23.597632 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:23.597605 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:23.980877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:23.980795 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:46:23.986922 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:46:23.986894 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f98bd5a_55a2_46ee_86b6_d13ea60af7fb.slice/crio-0c874db6e09d42f5dd6e1b9d786fd887b4cc576b836e5395c308daaab82d4a46 WatchSource:0}: Error finding container 0c874db6e09d42f5dd6e1b9d786fd887b4cc576b836e5395c308daaab82d4a46: Status 404 returned error can't find the container with id 0c874db6e09d42f5dd6e1b9d786fd887b4cc576b836e5395c308daaab82d4a46 Nov 02 06:46:24.081717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:24.081681 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" event={"ID":"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb","Type":"ContainerStarted","Data":"0c874db6e09d42f5dd6e1b9d786fd887b4cc576b836e5395c308daaab82d4a46"} Nov 02 06:46:26.098638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.098584 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" event={"ID":"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb","Type":"ContainerStarted","Data":"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69"} Nov 02 06:46:26.099088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.098693 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:26.217291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.217241 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" podStartSLOduration=3.137808037 podStartE2EDuration="4.217227749s" totalImagesPullingTime="1.079419712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:26.21442507 +0000 UTC m=+1740.187759754" watchObservedRunningTime="2026-11-02 06:46:26.217227749 +0000 UTC m=+1740.190562433" Nov 02 06:46:26.421136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.421104 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:46:26.425710 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.425691 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.498852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.498811 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e233a7\"" Nov 02 06:46:26.499438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.499415 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e233a7-dockercfg-dmb68\"" Nov 02 06:46:26.521938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.521898 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:46:26.539848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.539813 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz6v\" (UniqueName: \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.540076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.539870 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.589377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.589342 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:46:26.599122 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.598684 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.635645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.635617 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:46:26.642102 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.642047 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6sb\" (UniqueName: \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.644276 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.644250 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz6v\" (UniqueName: \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.644437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.644423 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.644572 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.644560 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.646296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.646274 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.699805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.699723 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz6v\" (UniqueName: \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\") pod \"payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.741014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.740945 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:26.745779 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.745725 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.745893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.745880 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6sb\" (UniqueName: \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.746451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.746429 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.802950 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.802732 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6sb\" (UniqueName: \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\") pod \"payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:26.927181 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:26.927148 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:27.133362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:27.133328 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:46:27.137982 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:46:27.137955 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35ea7e6_30fa_43b2_87b5_d4bf469ed961.slice/crio-9ed4ce70be56654d499dc268d9ab1bcb09fa4bc15607833b1eb10eb9a25c88d0 WatchSource:0}: Error finding container 9ed4ce70be56654d499dc268d9ab1bcb09fa4bc15607833b1eb10eb9a25c88d0: Status 404 returned error can't find the container with id 9ed4ce70be56654d499dc268d9ab1bcb09fa4bc15607833b1eb10eb9a25c88d0 Nov 02 06:46:27.309188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:27.309114 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:46:27.315175 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:46:27.315150 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ee9498_d9b9_47bd_8246_ec5d82c79605.slice/crio-6989f31da6e379229d87dac920e55b1024e093fcb7d06ce5aafaa50e7e245e00 WatchSource:0}: Error finding container 6989f31da6e379229d87dac920e55b1024e093fcb7d06ce5aafaa50e7e245e00: Status 404 returned error can't find the container with id 6989f31da6e379229d87dac920e55b1024e093fcb7d06ce5aafaa50e7e245e00 Nov 02 06:46:28.115362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.115324 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" event={"ID":"13ee9498-d9b9-47bd-8246-ec5d82c79605","Type":"ContainerStarted","Data":"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8"} Nov 02 06:46:28.115615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.115392 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:46:28.115615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.115416 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" event={"ID":"13ee9498-d9b9-47bd-8246-ec5d82c79605","Type":"ContainerStarted","Data":"6989f31da6e379229d87dac920e55b1024e093fcb7d06ce5aafaa50e7e245e00"} Nov 02 06:46:28.117038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.117011 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" event={"ID":"d35ea7e6-30fa-43b2-87b5-d4bf469ed961","Type":"ContainerStarted","Data":"febb4bea916890eca44ec573fcd2cee3f6f8d571c515d5e1dda63e6729288c88"} Nov 02 06:46:28.117132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.117054 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" event={"ID":"d35ea7e6-30fa-43b2-87b5-d4bf469ed961","Type":"ContainerStarted","Data":"9ed4ce70be56654d499dc268d9ab1bcb09fa4bc15607833b1eb10eb9a25c88d0"} Nov 02 06:46:28.117132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.117075 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:28.225094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.225043 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" podStartSLOduration=2.225029174 podStartE2EDuration="2.225029174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:28.222111507 +0000 UTC m=+1742.195446214" watchObservedRunningTime="2026-11-02 06:46:28.225029174 +0000 UTC m=+1742.198363898" Nov 02 06:46:28.329431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:28.329348 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" podStartSLOduration=2.329332224 podStartE2EDuration="2.329332224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:28.328589083 +0000 UTC m=+1742.301923766" watchObservedRunningTime="2026-11-02 06:46:28.329332224 +0000 UTC m=+1742.302666909" Nov 02 06:46:32.112308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:32.112280 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:46:38.554590 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:46:38.554551 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:46:39.126944 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:39.126908 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:46:39.127168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:46:39.127000 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:47:00.207410 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:00.207377 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:47:06.106267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:06.106233 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:47:06.106833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:06.106459 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" containerID="cri-o://d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f" gracePeriod=30 Nov 02 06:47:08.553716 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:47:08.553662 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:47:09.893624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:09.893584 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:47:09.894024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:09.893654 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:47:11.388159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.388099 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:47:11.449509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449476 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449570 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449595 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449670 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449746 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.449795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449756 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449818 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.449951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.449861 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.450112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450046 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.450112 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450082 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:11.450218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450111 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\") pod \"75537f40-6928-4c3a-b0ab-5879184a9d7e\" (UID: \"75537f40-6928-4c3a-b0ab-5879184a9d7e\") " Nov 02 06:47:11.450269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450218 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.450350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450304 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.450817 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450798 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.450959 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450798 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.451115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.450971 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.451115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.451012 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.451115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.451028 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75537f40-6928-4c3a-b0ab-5879184a9d7e-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.452468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.452426 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:11.452468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.452446 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:11.452639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.452610 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "kube-api-access-8zcs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:11.452699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.452657 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy" pod "75537f40-6928-4c3a-b0ab-5879184a9d7e" (UID: "75537f40-6928-4c3a-b0ab-5879184a9d7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.457758 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.457729 6273 generic.go:356] "Generic (PLEG): container finished" podID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerID="d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f" exitCode=0 Nov 02 06:47:11.457852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.457773 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" Nov 02 06:47:11.457852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.457813 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" event={"ID":"75537f40-6928-4c3a-b0ab-5879184a9d7e","Type":"ContainerDied","Data":"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f"} Nov 02 06:47:11.457935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.457870 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf" event={"ID":"75537f40-6928-4c3a-b0ab-5879184a9d7e","Type":"ContainerDied","Data":"eb04f317dc58d62aae3142defc7a7afe0f27dd12ee109c5839fb71e7f4cb637b"} Nov 02 06:47:11.457935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.457900 6273 scope.go:122] "RemoveContainer" containerID="d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f" Nov 02 06:47:11.483522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.483503 6273 scope.go:122] "RemoveContainer" containerID="d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f" Nov 02 06:47:11.483836 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:47:11.483814 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f\": container with ID starting with d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f not found: ID does not exist" containerID="d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f" Nov 02 06:47:11.483924 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.483846 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f"} err="failed to get container status \"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f\": rpc error: code = NotFound desc = could not find container \"d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f\": container with ID starting with d3bde10a34210780153568ff47ac95023d3f737a9dbe0316ae0f46a103c8324f not found: ID does not exist" Nov 02 06:47:11.552019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.551952 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcs8\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-kube-api-access-8zcs8\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.552019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.552000 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.552019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.552016 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.552212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.552028 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.552212 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.552039 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75537f40-6928-4c3a-b0ab-5879184a9d7e-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.659957 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.659930 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:47:11.665363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:11.665334 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gd6wf"] Nov 02 06:47:12.183555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:12.183519 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" path="/var/lib/kubelet/pods/75537f40-6928-4c3a-b0ab-5879184a9d7e/volumes" Nov 02 06:47:32.432867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.432835 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:47:32.433616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.433596 6273 cpu_manager.go:414] "containerMap: removing container" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" Nov 02 06:47:32.433616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.433614 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" Nov 02 06:47:32.433784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.433775 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="75537f40-6928-4c3a-b0ab-5879184a9d7e" containerName="istio-proxy" Nov 02 06:47:32.439265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.439246 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.491013 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.490973 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:47:32.515805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.515777 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkg4\" (UniqueName: \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.515946 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.515833 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.516090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.516066 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.516158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.516127 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.534395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.534367 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6d13c3-serving-cert\"" Nov 02 06:47:32.617515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.617485 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkg4\" (UniqueName: \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.617705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.617546 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.617705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.617658 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.617705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.617694 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.617895 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:47:32.617874 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-serving-cert: secret "maas-api-e2e-shared-b-w2-6d13c3-serving-cert" not found Nov 02 06:47:32.617969 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:47:32.617955 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls podName:dfecce48-759e-4b7d-85bc-d834ec21d0a6 nodeName:}" failed. No retries permitted until 2026-11-02 06:47:33.117933234 +0000 UTC m=+1807.091267913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" (UID: "dfecce48-759e-4b7d-85bc-d834ec21d0a6") : secret "maas-api-e2e-shared-b-w2-6d13c3-serving-cert" not found Nov 02 06:47:32.618302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.618284 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.619899 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.619880 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:32.673582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:32.673554 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkg4\" (UniqueName: \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:33.125229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:33.125196 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:33.127641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:33.127613 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:33.353217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:33.353184 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:33.732338 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:33.732259 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:47:33.738492 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:47:33.738447 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfecce48_759e_4b7d_85bc_d834ec21d0a6.slice/crio-ec857e51285f6ad3c8d5727028ec817708c217198b44549ae4934563acc98e30 WatchSource:0}: Error finding container ec857e51285f6ad3c8d5727028ec817708c217198b44549ae4934563acc98e30: Status 404 returned error can't find the container with id ec857e51285f6ad3c8d5727028ec817708c217198b44549ae4934563acc98e30 Nov 02 06:47:34.631144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:34.631113 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" event={"ID":"dfecce48-759e-4b7d-85bc-d834ec21d0a6","Type":"ContainerStarted","Data":"ec857e51285f6ad3c8d5727028ec817708c217198b44549ae4934563acc98e30"} Nov 02 06:47:35.639744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:35.639707 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" event={"ID":"dfecce48-759e-4b7d-85bc-d834ec21d0a6","Type":"ContainerStarted","Data":"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d"} Nov 02 06:47:35.640271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:35.639775 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:47:35.752776 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:35.752720 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" podStartSLOduration=2.874258578 podStartE2EDuration="3.75270458s" totalImagesPullingTime="878.446002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:47:35.749443137 +0000 UTC m=+1809.722777828" watchObservedRunningTime="2026-11-02 06:47:35.75270458 +0000 UTC m=+1809.726039265" Nov 02 06:47:38.553577 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:47:38.553534 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:47:41.668065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:47:41.668029 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:48:06.493524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:06.493488 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:48:08.555190 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:08.555159 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:48:18.951015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:18.950964 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:48:19.051786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:19.051757 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:48:23.952240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:23.952198 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" podUID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" containerName="payload-processing" containerID="cri-o://f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2" gracePeriod=25 Nov 02 06:48:24.052771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.052717 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" podUID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" containerName="payload-pre-processing" containerID="cri-o://4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6" gracePeriod=25 Nov 02 06:48:24.247188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.246816 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:48:24.291854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.291822 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\") pod \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " Nov 02 06:48:24.292149 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.292125 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\" (UniqueName: \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\") pod \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\" (UID: \"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee\") " Nov 02 06:48:24.292287 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.292264 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume" pod "8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" (UID: "8463ed4e-7fbe-4929-9bb8-34a4af15f8ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:24.293245 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.293216 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:24.294968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.294936 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42" pod "8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" (UID: "8463ed4e-7fbe-4929-9bb8-34a4af15f8ee"). InnerVolumeSpecName "kube-api-access-jdx42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:24.343094 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.343070 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:48:24.394325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.394291 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\") pod \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " Nov 02 06:48:24.394569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.394539 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\" (UniqueName: \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\") pod \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\" (UID: \"4ced5489-3232-4eb6-8cf9-36977e67c1cd\") " Nov 02 06:48:24.394724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.394696 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume" pod "4ced5489-3232-4eb6-8cf9-36977e67c1cd" (UID: "4ced5489-3232-4eb6-8cf9-36977e67c1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:24.395406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.395332 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdx42\" (UniqueName: \"kubernetes.io/projected/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee-kube-api-access-jdx42\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:24.395406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.395363 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ced5489-3232-4eb6-8cf9-36977e67c1cd-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:24.396876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.396847 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7" pod "4ced5489-3232-4eb6-8cf9-36977e67c1cd" (UID: "4ced5489-3232-4eb6-8cf9-36977e67c1cd"). InnerVolumeSpecName "kube-api-access-v7zn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:24.496935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:24.496858 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zn7\" (UniqueName: \"kubernetes.io/projected/4ced5489-3232-4eb6-8cf9-36977e67c1cd-kube-api-access-v7zn7\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:25.051382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.051346 6273 generic.go:356] "Generic (PLEG): container finished" podID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" containerID="f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2" exitCode=0 Nov 02 06:48:25.051884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.051398 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" Nov 02 06:48:25.051884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.051432 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" event={"ID":"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee","Type":"ContainerDied","Data":"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2"} Nov 02 06:48:25.051884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.051482 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6" event={"ID":"8463ed4e-7fbe-4929-9bb8-34a4af15f8ee","Type":"ContainerDied","Data":"b753811720e94ae9be1b601c0dc5ef2d12bc7da633b656b7437d53e276ca05a5"} Nov 02 06:48:25.051884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.051499 6273 scope.go:122] "RemoveContainer" containerID="f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2" Nov 02 06:48:25.053074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.053050 6273 generic.go:356] "Generic (PLEG): container finished" podID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" containerID="4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6" exitCode=0 Nov 02 06:48:25.053153 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.053101 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" Nov 02 06:48:25.053153 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.053123 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" event={"ID":"4ced5489-3232-4eb6-8cf9-36977e67c1cd","Type":"ContainerDied","Data":"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6"} Nov 02 06:48:25.053240 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.053176 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t" event={"ID":"4ced5489-3232-4eb6-8cf9-36977e67c1cd","Type":"ContainerDied","Data":"7362c20b3935295ede45a49d73a385402a7542ff40569bbfe094d3c1e93b7a8f"} Nov 02 06:48:25.062640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.062622 6273 scope.go:122] "RemoveContainer" containerID="f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2" Nov 02 06:48:25.063024 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:25.062978 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2\": container with ID starting with f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2 not found: ID does not exist" containerID="f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2" Nov 02 06:48:25.063146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.063027 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2"} err="failed to get container status \"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2\": rpc error: code = NotFound desc = could not find container \"f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2\": container with ID starting with f5f8011dcf14bccbf13d16254008827e66d9273e04e58dbd22adc6927164bee2 not found: ID does not exist" Nov 02 06:48:25.063146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.063052 6273 scope.go:122] "RemoveContainer" containerID="4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6" Nov 02 06:48:25.076279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.076255 6273 scope.go:122] "RemoveContainer" containerID="4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6" Nov 02 06:48:25.076615 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:25.076593 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6\": container with ID starting with 4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6 not found: ID does not exist" containerID="4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6" Nov 02 06:48:25.076671 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.076622 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6"} err="failed to get container status \"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6\": rpc error: code = NotFound desc = could not find container \"4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6\": container with ID starting with 4618820bdb6d2b0064f239590d6232a4ff3d3f2b08b68054753494f03bc799c6 not found: ID does not exist" Nov 02 06:48:25.206689 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.206651 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:48:25.252808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.252716 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12bfb5-7b59695897-lwcp6"] Nov 02 06:48:25.312666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.312631 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:48:25.315763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:25.315738 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12bfb5-77594f7fd7-j6g9t"] Nov 02 06:48:26.181839 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:26.181809 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" path="/var/lib/kubelet/pods/4ced5489-3232-4eb6-8cf9-36977e67c1cd/volumes" Nov 02 06:48:26.182303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:26.182283 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" path="/var/lib/kubelet/pods/8463ed4e-7fbe-4929-9bb8-34a4af15f8ee/volumes" Nov 02 06:48:26.957562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:26.957531 6273 scope.go:122] "RemoveContainer" containerID="846e36c0671e50fe2087c2017fbeb4eb0ce4fdee8ad6239cbfd05dcd40d7368b" Nov 02 06:48:26.968440 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:26.968421 6273 scope.go:122] "RemoveContainer" containerID="5da0c49ab5a4e6e12d6f8a65570671a94805fb1e2051d0480c2b67ddbe703df3" Nov 02 06:48:37.599401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:37.599362 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:48:37.599890 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:37.599651 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" containerID="cri-o://0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89" gracePeriod=30 Nov 02 06:48:38.553664 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:38.553630 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:48:42.876931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.876862 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:48:42.960632 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.960585 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.960801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.960676 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.960801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.960785 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.960915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.960822 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.960915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.960871 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.961052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961027 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.961121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961081 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.961121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961100 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:42.961333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961292 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:42.961447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961420 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:42.961514 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961480 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.961607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961589 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\") pod \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\" (UID: \"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472\") " Nov 02 06:48:42.961865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.961739 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:42.962075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962047 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:42.962633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962607 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.962748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962639 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.962748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962655 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.962748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962671 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.962748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.962686 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.963138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.963111 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:42.963256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.963229 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "kube-api-access-phwsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:42.963679 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.963653 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:42.963759 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:42.963656 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy" pod "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" (UID: "6d3e6a8b-f7df-4153-91c1-5c64e0bb1472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:43.065142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.065104 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:43.065142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.065143 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwsx\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-kube-api-access-phwsx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:43.065332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.065160 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:43.065332 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.065173 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:43.189159 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.189130 6273 generic.go:356] "Generic (PLEG): container finished" podID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerID="0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89" exitCode=0 Nov 02 06:48:43.189335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.189165 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" Nov 02 06:48:43.189335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.189210 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" event={"ID":"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472","Type":"ContainerDied","Data":"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89"} Nov 02 06:48:43.189335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.189250 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8" event={"ID":"6d3e6a8b-f7df-4153-91c1-5c64e0bb1472","Type":"ContainerDied","Data":"78453388b6bb4005fdb79a5aaf6d6c0e2794e8902d6d7fd9b4b57d5545a0d59c"} Nov 02 06:48:43.189335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.189273 6273 scope.go:122] "RemoveContainer" containerID="0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89" Nov 02 06:48:43.200080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.200065 6273 scope.go:122] "RemoveContainer" containerID="0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89" Nov 02 06:48:43.200330 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:43.200309 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89\": container with ID starting with 0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89 not found: ID does not exist" containerID="0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89" Nov 02 06:48:43.200409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.200336 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89"} err="failed to get container status \"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89\": rpc error: code = NotFound desc = could not find container \"0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89\": container with ID starting with 0bd18d4c6bf66f16f0dddc421939b9c2178e06eee40f2fe07e3dd76acae88a89 not found: ID does not exist" Nov 02 06:48:43.345015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.344975 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:48:43.349477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:43.349442 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-pl8s8"] Nov 02 06:48:44.181355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:44.181322 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" path="/var/lib/kubelet/pods/6d3e6a8b-f7df-4153-91c1-5c64e0bb1472/volumes" Nov 02 06:48:46.104695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:46.104659 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:48:51.289126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:51.289086 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:48:51.289745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:51.289348 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" containerID="cri-o://1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7" gracePeriod=30 Nov 02 06:48:51.582352 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:51.582255 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:48:51.582670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:51.582500 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" containerID="cri-o://02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d" gracePeriod=30 Nov 02 06:48:53.117462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:53.117378 6273 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:48:53.117462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:53.117440 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:48:56.556343 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.556283 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:48:56.664548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664509 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.664706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664573 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.664706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664618 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.664824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664707 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.664824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664784 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.664948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664824 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.664948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.664920 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.665114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665017 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.665114 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665042 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.665327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665236 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.665327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665245 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.665327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665303 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\") pod \"453e9a62-204d-4b39-8dd8-ca59f9235996\" (UID: \"453e9a62-204d-4b39-8dd8-ca59f9235996\") " Nov 02 06:48:56.665597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665412 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.665735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.665707 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:56.666104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.666081 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.666207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.666109 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.666207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.666124 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.666207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.666139 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453e9a62-204d-4b39-8dd8-ca59f9235996-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.666207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.666154 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.667201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.667171 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.667296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.667208 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "kube-api-access-j98v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.667296 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.667241 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:56.667735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.667716 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy" pod "453e9a62-204d-4b39-8dd8-ca59f9235996" (UID: "453e9a62-204d-4b39-8dd8-ca59f9235996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.768197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.768148 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j98v2\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-kube-api-access-j98v2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.768197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.768190 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.768197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.768205 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.768499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.768219 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453e9a62-204d-4b39-8dd8-ca59f9235996-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.863155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.863129 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:48:56.970066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970029 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970122 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970152 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970234 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970311 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970311 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.970418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970375 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.970418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970414 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970472 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970506 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.970628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970521 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970525 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.970628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970594 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\") pod \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\" (UID: \"f0bfb9b4-8fad-443c-9889-7c50080f6a50\") " Nov 02 06:48:56.970864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.970788 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:56.971135 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.971111 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.971135 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.971136 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.971351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.971149 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.971351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.971164 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.971351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.971177 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.973203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.973140 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.973203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.973164 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "kube-api-access-hrr9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.973824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.973797 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:56.974413 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:56.974389 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy" pod "f0bfb9b4-8fad-443c-9889-7c50080f6a50" (UID: "f0bfb9b4-8fad-443c-9889-7c50080f6a50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.072318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.072280 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.072318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.072312 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.072523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.072326 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/f0bfb9b4-8fad-443c-9889-7c50080f6a50-kube-api-access-hrr9q\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.072523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.072342 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bfb9b4-8fad-443c-9889-7c50080f6a50-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.252787 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.252746 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:48:57.252977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.252955 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerName="discovery" containerID="cri-o://a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329" gracePeriod=30 Nov 02 06:48:57.291433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.291404 6273 generic.go:356] "Generic (PLEG): container finished" podID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerID="02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d" exitCode=0 Nov 02 06:48:57.291570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.291450 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" Nov 02 06:48:57.291570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.291478 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" event={"ID":"f0bfb9b4-8fad-443c-9889-7c50080f6a50","Type":"ContainerDied","Data":"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d"} Nov 02 06:48:57.291570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.291532 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz" event={"ID":"f0bfb9b4-8fad-443c-9889-7c50080f6a50","Type":"ContainerDied","Data":"e93280bf0345d9d1fae1bb7a4a7b6e2e4d52ff2d1ea82c874acdba7a53bab8d7"} Nov 02 06:48:57.291570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.291553 6273 scope.go:122] "RemoveContainer" containerID="02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d" Nov 02 06:48:57.293054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.292973 6273 generic.go:356] "Generic (PLEG): container finished" podID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerID="1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7" exitCode=0 Nov 02 06:48:57.293054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.293035 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" event={"ID":"453e9a62-204d-4b39-8dd8-ca59f9235996","Type":"ContainerDied","Data":"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7"} Nov 02 06:48:57.293054 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.293050 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" Nov 02 06:48:57.293232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.293075 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw" event={"ID":"453e9a62-204d-4b39-8dd8-ca59f9235996","Type":"ContainerDied","Data":"b5dcdd36f89bbd46362babba7e269392ad24c2b60a9d718e7f58f18c9db418e6"} Nov 02 06:48:57.369096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.369072 6273 scope.go:122] "RemoveContainer" containerID="02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d" Nov 02 06:48:57.369468 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:57.369441 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d\": container with ID starting with 02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d not found: ID does not exist" containerID="02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d" Nov 02 06:48:57.369589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.369479 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d"} err="failed to get container status \"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d\": rpc error: code = NotFound desc = could not find container \"02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d\": container with ID starting with 02f9ee78ca3d5ec410a1b78a091e416e1c11b10f0b02b78750138a759a52448d not found: ID does not exist" Nov 02 06:48:57.369589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.369501 6273 scope.go:122] "RemoveContainer" containerID="1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7" Nov 02 06:48:57.389911 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.389810 6273 scope.go:122] "RemoveContainer" containerID="1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7" Nov 02 06:48:57.390218 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:57.390196 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7\": container with ID starting with 1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7 not found: ID does not exist" containerID="1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7" Nov 02 06:48:57.390289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.390229 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7"} err="failed to get container status \"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7\": rpc error: code = NotFound desc = could not find container \"1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7\": container with ID starting with 1e8e7fb849e6bbfe065150a7159b57eec76daaaf032a9af4f140168f1981c5a7 not found: ID does not exist" Nov 02 06:48:57.476196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.476160 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:48:57.479615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.479584 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-dc5fw"] Nov 02 06:48:57.552398 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.552374 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:48:57.579519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579489 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579557 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579678 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579733 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579789 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579850 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.579942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579902 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:57.580271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.579977 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\") pod \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\" (UID: \"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f\") " Nov 02 06:48:57.580583 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.580559 6273 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-ca-configmap\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.582132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.581948 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:57.582357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.582324 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:57.582357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.582346 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.582517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.582450 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:57.584057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.582858 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:57.584057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.582967 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr" pod "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" (UID: "cd78ba7d-c1f4-4cc5-813c-007f6d13a96f"). InnerVolumeSpecName "kube-api-access-j6qsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:57.584888 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.584863 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:48:57.589331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.589308 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-48cnz"] Nov 02 06:48:57.682200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682157 6273 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-csr-dns-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.682200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682184 6273 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-cacerts\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.682200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682197 6273 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-local-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.682200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682206 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.682542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682216 6273 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-istio-kubeconfig\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.682542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:57.682225 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qsr\" (UniqueName: \"kubernetes.io/projected/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f-kube-api-access-j6qsr\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.187288 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.187250 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" path="/var/lib/kubelet/pods/453e9a62-204d-4b39-8dd8-ca59f9235996/volumes" Nov 02 06:48:58.187960 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.187937 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" path="/var/lib/kubelet/pods/f0bfb9b4-8fad-443c-9889-7c50080f6a50/volumes" Nov 02 06:48:58.302311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.302270 6273 generic.go:356] "Generic (PLEG): container finished" podID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerID="a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329" exitCode=0 Nov 02 06:48:58.302505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.302332 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" Nov 02 06:48:58.302505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.302353 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" event={"ID":"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f","Type":"ContainerDied","Data":"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329"} Nov 02 06:48:58.302505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.302404 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll" event={"ID":"cd78ba7d-c1f4-4cc5-813c-007f6d13a96f","Type":"ContainerDied","Data":"79ed72b16921d22b053ceb8f1c7ec7d5e74a329c83a6d6e75dac3d00ba5cc60e"} Nov 02 06:48:58.302505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.302433 6273 scope.go:122] "RemoveContainer" containerID="a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329" Nov 02 06:48:58.313115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.313097 6273 scope.go:122] "RemoveContainer" containerID="a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329" Nov 02 06:48:58.313392 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:48:58.313364 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329\": container with ID starting with a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329 not found: ID does not exist" containerID="a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329" Nov 02 06:48:58.313467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.313403 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329"} err="failed to get container status \"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329\": rpc error: code = NotFound desc = could not find container \"a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329\": container with ID starting with a4122c4df75b90ba5a67a6a57096b7d8fc15664de1c89245af015c2143380329 not found: ID does not exist" Nov 02 06:48:58.458050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.457957 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:48:58.503948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:48:58.503917 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4dqll"] Nov 02 06:49:00.181234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:00.181200 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" path="/var/lib/kubelet/pods/cd78ba7d-c1f4-4cc5-813c-007f6d13a96f/volumes" Nov 02 06:49:08.555013 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:08.554903 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:49:20.667260 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.667217 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:49:20.765478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.765443 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:49:20.972670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.972590 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:49:20.973502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973476 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerName="discovery" Nov 02 06:49:20.973502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973499 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerName="discovery" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973518 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973524 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973555 6273 cpu_manager.go:414] "containerMap: removing container" podUID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" containerName="payload-pre-processing" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973560 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" containerName="payload-pre-processing" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973570 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" containerName="payload-processing" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973575 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" containerName="payload-processing" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973596 6273 cpu_manager.go:414] "containerMap: removing container" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973601 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973610 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973614 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973707 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="453e9a62-204d-4b39-8dd8-ca59f9235996" containerName="istio-proxy" Nov 02 06:49:20.973711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973717 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bfb9b4-8fad-443c-9889-7c50080f6a50" containerName="istio-proxy" Nov 02 06:49:20.974327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973723 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd78ba7d-c1f4-4cc5-813c-007f6d13a96f" containerName="discovery" Nov 02 06:49:20.974327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973730 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3e6a8b-f7df-4153-91c1-5c64e0bb1472" containerName="istio-proxy" Nov 02 06:49:20.974327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973743 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ced5489-3232-4eb6-8cf9-36977e67c1cd" containerName="payload-pre-processing" Nov 02 06:49:20.974327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.973750 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8463ed4e-7fbe-4929-9bb8-34a4af15f8ee" containerName="payload-processing" Nov 02 06:49:20.977359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:20.977339 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.030375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.030351 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:49:21.051166 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.040592 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3bf02ebf-serving-cert\"" Nov 02 06:49:21.054201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.054162 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.054422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.054404 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.054535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.054473 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4p7x\" (UniqueName: \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.054617 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.054592 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.155640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.155599 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.155854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.155692 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.155854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.155729 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4p7x\" (UniqueName: \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.155854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.155772 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.156060 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:21.155946 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-serving-cert: secret "maas-api-e2e-mt-3bf02ebf-serving-cert" not found Nov 02 06:49:21.156129 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:21.156105 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls podName:b59768cf-940c-4121-b475-7e7b718d8e3a nodeName:}" failed. No retries permitted until 2026-11-02 06:49:21.656082776 +0000 UTC m=+1915.629417438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls") pod "maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" (UID: "b59768cf-940c-4121-b475-7e7b718d8e3a") : secret "maas-api-e2e-mt-3bf02ebf-serving-cert" not found Nov 02 06:49:21.156433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.156404 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.158379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.158358 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.216722 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.216693 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4p7x\" (UniqueName: \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.663613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.663579 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.665931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.665905 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:21.891177 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:21.891146 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:22.266302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:22.266219 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:49:22.272685 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:49:22.272655 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59768cf_940c_4121_b475_7e7b718d8e3a.slice/crio-eda05d86264b356c919bf992867e3b9d75c8586640557eb5ba9db834ce383f74 WatchSource:0}: Error finding container eda05d86264b356c919bf992867e3b9d75c8586640557eb5ba9db834ce383f74: Status 404 returned error can't find the container with id eda05d86264b356c919bf992867e3b9d75c8586640557eb5ba9db834ce383f74 Nov 02 06:49:22.274239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:22.274218 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:49:22.508929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:22.508893 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" event={"ID":"b59768cf-940c-4121-b475-7e7b718d8e3a","Type":"ContainerStarted","Data":"eda05d86264b356c919bf992867e3b9d75c8586640557eb5ba9db834ce383f74"} Nov 02 06:49:23.518308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:23.518216 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" event={"ID":"b59768cf-940c-4121-b475-7e7b718d8e3a","Type":"ContainerStarted","Data":"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e"} Nov 02 06:49:23.518703 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:23.518416 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:23.630971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:23.630917 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" podStartSLOduration=2.652168448 podStartE2EDuration="3.630903313s" totalImagesPullingTime="978.734865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:49:23.628252621 +0000 UTC m=+1917.601587308" watchObservedRunningTime="2026-11-02 06:49:23.630903313 +0000 UTC m=+1917.604238036" Nov 02 06:49:25.667610 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:25.667570 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" podUID="111b1c1f-09c7-482e-b915-2a1cdb967a70" containerName="payload-processing" containerID="cri-o://3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b" gracePeriod=25 Nov 02 06:49:25.766606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:25.766569 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" podUID="089de147-27e1-467f-bfbd-753ab389ca0e" containerName="payload-pre-processing" containerID="cri-o://8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7" gracePeriod=25 Nov 02 06:49:25.941629 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:25.941562 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:49:26.036208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.036182 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\") pod \"111b1c1f-09c7-482e-b915-2a1cdb967a70\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " Nov 02 06:49:26.036379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.036315 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\" (UniqueName: \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\") pod \"111b1c1f-09c7-482e-b915-2a1cdb967a70\" (UID: \"111b1c1f-09c7-482e-b915-2a1cdb967a70\") " Nov 02 06:49:26.036698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.036669 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume" pod "111b1c1f-09c7-482e-b915-2a1cdb967a70" (UID: "111b1c1f-09c7-482e-b915-2a1cdb967a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:26.036959 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.036938 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b1c1f-09c7-482e-b915-2a1cdb967a70-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:26.038496 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.038472 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt" pod "111b1c1f-09c7-482e-b915-2a1cdb967a70" (UID: "111b1c1f-09c7-482e-b915-2a1cdb967a70"). InnerVolumeSpecName "kube-api-access-vn4vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:26.048968 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.048946 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:49:26.138072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.138045 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4vt\" (UniqueName: \"kubernetes.io/projected/111b1c1f-09c7-482e-b915-2a1cdb967a70-kube-api-access-vn4vt\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:26.239253 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.239222 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\") pod \"089de147-27e1-467f-bfbd-753ab389ca0e\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " Nov 02 06:49:26.239386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.239354 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\" (UniqueName: \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\") pod \"089de147-27e1-467f-bfbd-753ab389ca0e\" (UID: \"089de147-27e1-467f-bfbd-753ab389ca0e\") " Nov 02 06:49:26.239591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.239547 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume" pod "089de147-27e1-467f-bfbd-753ab389ca0e" (UID: "089de147-27e1-467f-bfbd-753ab389ca0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:26.240144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.240123 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089de147-27e1-467f-bfbd-753ab389ca0e-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:26.241227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.241206 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg" pod "089de147-27e1-467f-bfbd-753ab389ca0e" (UID: "089de147-27e1-467f-bfbd-753ab389ca0e"). InnerVolumeSpecName "kube-api-access-vz4dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:26.342154 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.342124 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4dg\" (UniqueName: \"kubernetes.io/projected/089de147-27e1-467f-bfbd-753ab389ca0e-kube-api-access-vz4dg\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:26.541841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.541753 6273 generic.go:356] "Generic (PLEG): container finished" podID="111b1c1f-09c7-482e-b915-2a1cdb967a70" containerID="3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b" exitCode=0 Nov 02 06:49:26.541841 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.541795 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" Nov 02 06:49:26.542084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.541844 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" event={"ID":"111b1c1f-09c7-482e-b915-2a1cdb967a70","Type":"ContainerDied","Data":"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b"} Nov 02 06:49:26.542084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.541898 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb" event={"ID":"111b1c1f-09c7-482e-b915-2a1cdb967a70","Type":"ContainerDied","Data":"1f2ecd7176ad93e7098b32ebb529cddf98f6f80543dc8d8f500c179d5c818c7a"} Nov 02 06:49:26.542084 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.541921 6273 scope.go:122] "RemoveContainer" containerID="3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b" Nov 02 06:49:26.543575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.543551 6273 generic.go:356] "Generic (PLEG): container finished" podID="089de147-27e1-467f-bfbd-753ab389ca0e" containerID="8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7" exitCode=0 Nov 02 06:49:26.543718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.543602 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" Nov 02 06:49:26.543718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.543629 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" event={"ID":"089de147-27e1-467f-bfbd-753ab389ca0e","Type":"ContainerDied","Data":"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7"} Nov 02 06:49:26.543718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.543685 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj" event={"ID":"089de147-27e1-467f-bfbd-753ab389ca0e","Type":"ContainerDied","Data":"6772b12451d91bceadb440f898a64d04daa1e26d85cf02bd6be1dcf4ff4d63e1"} Nov 02 06:49:26.552788 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.552764 6273 scope.go:122] "RemoveContainer" containerID="3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b" Nov 02 06:49:26.553077 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:26.553054 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b\": container with ID starting with 3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b not found: ID does not exist" containerID="3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b" Nov 02 06:49:26.553136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.553086 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b"} err="failed to get container status \"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b\": rpc error: code = NotFound desc = could not find container \"3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b\": container with ID starting with 3f0704f4b06fd3a1b949a9a230621743b299a3bcfa4e242cc2cc90d818b5604b not found: ID does not exist" Nov 02 06:49:26.553136 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.553103 6273 scope.go:122] "RemoveContainer" containerID="8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7" Nov 02 06:49:26.563259 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.563234 6273 scope.go:122] "RemoveContainer" containerID="8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7" Nov 02 06:49:26.563524 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:26.563502 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7\": container with ID starting with 8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7 not found: ID does not exist" containerID="8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7" Nov 02 06:49:26.563597 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.563532 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7"} err="failed to get container status \"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7\": rpc error: code = NotFound desc = could not find container \"8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7\": container with ID starting with 8444355fd4b80689ee28919acfb68f43214b98eaac51c13e934ff816628584a7 not found: ID does not exist" Nov 02 06:49:26.697382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.697346 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:49:26.700382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.700358 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-08bbf9-548d9fccbc-k6fgb"] Nov 02 06:49:26.803557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.803484 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:49:26.812996 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:26.812956 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-08bbf9-685dbf54bc-nhxqj"] Nov 02 06:49:28.181038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:28.180983 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089de147-27e1-467f-bfbd-753ab389ca0e" path="/var/lib/kubelet/pods/089de147-27e1-467f-bfbd-753ab389ca0e/volumes" Nov 02 06:49:28.181584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:28.181561 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111b1c1f-09c7-482e-b915-2a1cdb967a70" path="/var/lib/kubelet/pods/111b1c1f-09c7-482e-b915-2a1cdb967a70/volumes" Nov 02 06:49:29.531424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:29.531397 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:49:35.593095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.593058 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:49:35.593591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.593329 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" podUID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" containerName="maas-api" containerID="cri-o://0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69" gracePeriod=30 Nov 02 06:49:35.901930 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.901895 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:49:35.960680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.960646 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") pod \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " Nov 02 06:49:35.960864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.960736 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\") pod \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " Nov 02 06:49:35.960864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.960833 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\" (UniqueName: \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\") pod \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " Nov 02 06:49:35.961082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.961055 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\") pod \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\" (UID: \"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb\") " Nov 02 06:49:35.961203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.961102 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca" pod "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" (UID: "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:35.961611 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.961589 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:35.962935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.962913 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h" pod "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" (UID: "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb"). InnerVolumeSpecName "kube-api-access-2r68h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:35.963276 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.963254 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls" pod "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" (UID: "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:49:35.963357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:35.963328 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs" pod "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" (UID: "2f98bd5a-55a2-46ee-86b6-d13ea60af7fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:49:36.063214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.063176 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:36.063214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.063215 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:36.063420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.063232 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r68h\" (UniqueName: \"kubernetes.io/projected/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb-kube-api-access-2r68h\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:36.617281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.617197 6273 generic.go:356] "Generic (PLEG): container finished" podID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" containerID="0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69" exitCode=0 Nov 02 06:49:36.617281 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.617251 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" Nov 02 06:49:36.617738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.617275 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" event={"ID":"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb","Type":"ContainerDied","Data":"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69"} Nov 02 06:49:36.617738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.617326 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s" event={"ID":"2f98bd5a-55a2-46ee-86b6-d13ea60af7fb","Type":"ContainerDied","Data":"0c874db6e09d42f5dd6e1b9d786fd887b4cc576b836e5395c308daaab82d4a46"} Nov 02 06:49:36.617738 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.617349 6273 scope.go:122] "RemoveContainer" containerID="0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69" Nov 02 06:49:36.627096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.627079 6273 scope.go:122] "RemoveContainer" containerID="0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69" Nov 02 06:49:36.627389 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:36.627370 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69\": container with ID starting with 0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69 not found: ID does not exist" containerID="0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69" Nov 02 06:49:36.627457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.627399 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69"} err="failed to get container status \"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69\": rpc error: code = NotFound desc = could not find container \"0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69\": container with ID starting with 0572e7e99f9faa62969989edf9e0916d1a1898a3d8f0f0650879efddfb688d69 not found: ID does not exist" Nov 02 06:49:36.774975 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.774935 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:49:36.777999 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:36.777964 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12bfb5-5b8b59689d-4tt4s"] Nov 02 06:49:37.814582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:37.814547 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:49:38.181619 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:38.181586 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" path="/var/lib/kubelet/pods/2f98bd5a-55a2-46ee-86b6-d13ea60af7fb/volumes" Nov 02 06:49:38.555532 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:38.555443 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:49:42.815239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:42.815196 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" podUID="97d3b437-1421-4c3f-9d17-ed2d00649991" containerName="payload-processing" containerID="cri-o://8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034" gracePeriod=25 Nov 02 06:49:43.087604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.087580 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:49:43.147878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.147848 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\" (UniqueName: \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\") pod \"97d3b437-1421-4c3f-9d17-ed2d00649991\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " Nov 02 06:49:43.148038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.148019 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\") pod \"97d3b437-1421-4c3f-9d17-ed2d00649991\" (UID: \"97d3b437-1421-4c3f-9d17-ed2d00649991\") " Nov 02 06:49:43.148403 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.148375 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume" pod "97d3b437-1421-4c3f-9d17-ed2d00649991" (UID: "97d3b437-1421-4c3f-9d17-ed2d00649991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:43.148608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.148587 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d3b437-1421-4c3f-9d17-ed2d00649991-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:43.150389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.150358 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t" pod "97d3b437-1421-4c3f-9d17-ed2d00649991" (UID: "97d3b437-1421-4c3f-9d17-ed2d00649991"). InnerVolumeSpecName "kube-api-access-znf2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:43.249715 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.249685 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znf2t\" (UniqueName: \"kubernetes.io/projected/97d3b437-1421-4c3f-9d17-ed2d00649991-kube-api-access-znf2t\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:43.684289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.684247 6273 generic.go:356] "Generic (PLEG): container finished" podID="97d3b437-1421-4c3f-9d17-ed2d00649991" containerID="8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034" exitCode=0 Nov 02 06:49:43.684489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.684306 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" Nov 02 06:49:43.684489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.684332 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" event={"ID":"97d3b437-1421-4c3f-9d17-ed2d00649991","Type":"ContainerDied","Data":"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034"} Nov 02 06:49:43.684489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.684418 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r" event={"ID":"97d3b437-1421-4c3f-9d17-ed2d00649991","Type":"ContainerDied","Data":"c66fb97f334003fa4e8cdfa06c14d528d86f455f542b43e3ad7f93482f89e416"} Nov 02 06:49:43.684489 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.684448 6273 scope.go:122] "RemoveContainer" containerID="8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034" Nov 02 06:49:43.694551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.694533 6273 scope.go:122] "RemoveContainer" containerID="8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034" Nov 02 06:49:43.694803 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:49:43.694783 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034\": container with ID starting with 8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034 not found: ID does not exist" containerID="8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034" Nov 02 06:49:43.694865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.694814 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034"} err="failed to get container status \"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034\": rpc error: code = NotFound desc = could not find container \"8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034\": container with ID starting with 8814ad03d8750eaef0f8be934f37b923809627318b6ba561641d4420f470d034 not found: ID does not exist" Nov 02 06:49:43.837922 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.837893 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:49:43.843829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:43.843806 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70898f-78564c9687-rw28r"] Nov 02 06:49:44.180674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:44.180644 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d3b437-1421-4c3f-9d17-ed2d00649991" path="/var/lib/kubelet/pods/97d3b437-1421-4c3f-9d17-ed2d00649991/volumes" Nov 02 06:49:57.002469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:57.002429 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:49:58.976178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:49:58.976132 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:50:03.976940 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:03.976895 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" podUID="e02b12f5-2751-4469-8a4b-7b9c07d41776" containerName="payload-pre-processing" containerID="cri-o://0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295" gracePeriod=25 Nov 02 06:50:04.250866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.250798 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:50:04.357043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.357014 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\") pod \"e02b12f5-2751-4469-8a4b-7b9c07d41776\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " Nov 02 06:50:04.357361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.357329 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\" (UniqueName: \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\") pod \"e02b12f5-2751-4469-8a4b-7b9c07d41776\" (UID: \"e02b12f5-2751-4469-8a4b-7b9c07d41776\") " Nov 02 06:50:04.357462 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.357388 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume" pod "e02b12f5-2751-4469-8a4b-7b9c07d41776" (UID: "e02b12f5-2751-4469-8a4b-7b9c07d41776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:04.357844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.357823 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02b12f5-2751-4469-8a4b-7b9c07d41776-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:04.361167 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.361124 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4" pod "e02b12f5-2751-4469-8a4b-7b9c07d41776" (UID: "e02b12f5-2751-4469-8a4b-7b9c07d41776"). InnerVolumeSpecName "kube-api-access-8fzn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:04.458863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.458833 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzn4\" (UniqueName: \"kubernetes.io/projected/e02b12f5-2751-4469-8a4b-7b9c07d41776-kube-api-access-8fzn4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:04.825348 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.825312 6273 generic.go:356] "Generic (PLEG): container finished" podID="e02b12f5-2751-4469-8a4b-7b9c07d41776" containerID="0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295" exitCode=0 Nov 02 06:50:04.825518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.825359 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" Nov 02 06:50:04.825518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.825393 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" event={"ID":"e02b12f5-2751-4469-8a4b-7b9c07d41776","Type":"ContainerDied","Data":"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295"} Nov 02 06:50:04.825518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.825449 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k" event={"ID":"e02b12f5-2751-4469-8a4b-7b9c07d41776","Type":"ContainerDied","Data":"c97b1e865337ac2afc81795420c93626438bfbb2477cfc08a44e56892c2a8e74"} Nov 02 06:50:04.825518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.825475 6273 scope.go:122] "RemoveContainer" containerID="0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295" Nov 02 06:50:04.835289 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.835273 6273 scope.go:122] "RemoveContainer" containerID="0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295" Nov 02 06:50:04.835522 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:04.835504 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295\": container with ID starting with 0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295 not found: ID does not exist" containerID="0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295" Nov 02 06:50:04.835575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.835528 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295"} err="failed to get container status \"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295\": rpc error: code = NotFound desc = could not find container \"0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295\": container with ID starting with 0c30d346756b91b0b8e2818304547d0bac9f4aeb1456bcfdefdd081ab322a295 not found: ID does not exist" Nov 02 06:50:04.981330 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.981298 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:50:04.984685 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:04.984664 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a25b83-66df747c5f-fpt9k"] Nov 02 06:50:06.182321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:06.182290 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02b12f5-2751-4469-8a4b-7b9c07d41776" path="/var/lib/kubelet/pods/e02b12f5-2751-4469-8a4b-7b9c07d41776/volumes" Nov 02 06:50:08.555004 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:08.554949 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:50:16.722645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.722608 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 06:50:16.723435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723415 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" containerName="maas-api" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723437 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" containerName="maas-api" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723463 6273 cpu_manager.go:414] "containerMap: removing container" podUID="089de147-27e1-467f-bfbd-753ab389ca0e" containerName="payload-pre-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723471 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089de147-27e1-467f-bfbd-753ab389ca0e" containerName="payload-pre-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723486 6273 cpu_manager.go:414] "containerMap: removing container" podUID="111b1c1f-09c7-482e-b915-2a1cdb967a70" containerName="payload-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723493 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111b1c1f-09c7-482e-b915-2a1cdb967a70" containerName="payload-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723549 6273 cpu_manager.go:414] "containerMap: removing container" podUID="97d3b437-1421-4c3f-9d17-ed2d00649991" containerName="payload-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723556 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d3b437-1421-4c3f-9d17-ed2d00649991" containerName="payload-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723580 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e02b12f5-2751-4469-8a4b-7b9c07d41776" containerName="payload-pre-processing" Nov 02 06:50:16.723601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723587 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02b12f5-2751-4469-8a4b-7b9c07d41776" containerName="payload-pre-processing" Nov 02 06:50:16.724115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723792 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f98bd5a-55a2-46ee-86b6-d13ea60af7fb" containerName="maas-api" Nov 02 06:50:16.724115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723812 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="089de147-27e1-467f-bfbd-753ab389ca0e" containerName="payload-pre-processing" Nov 02 06:50:16.724115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723826 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="111b1c1f-09c7-482e-b915-2a1cdb967a70" containerName="payload-processing" Nov 02 06:50:16.724115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723844 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d3b437-1421-4c3f-9d17-ed2d00649991" containerName="payload-processing" Nov 02 06:50:16.724115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.723855 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02b12f5-2751-4469-8a4b-7b9c07d41776" containerName="payload-pre-processing" Nov 02 06:50:16.727382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.727362 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:16.823575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.823547 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a2ba11-serving-cert\"" Nov 02 06:50:16.826437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.826413 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 06:50:16.916747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.916707 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:16.916933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.916881 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj78\" (UniqueName: \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:16.917052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.917034 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:16.917110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:16.917076 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.017913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.017846 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.017913 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.017884 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.018142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.017975 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.018142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.018117 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj78\" (UniqueName: \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.018613 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.018591 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.020335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.020313 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.020451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.020434 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.071912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.071883 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj78\" (UniqueName: \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\") pod \"maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.341945 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.341859 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:17.722268 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.722232 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 06:50:17.726696 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:50:17.726668 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c50225_9ac4_4438_a308_6daf361948b5.slice/crio-e0a8d61d638cefe03681236daa93a3db008be374cd4092c221b13a1dd13df387 WatchSource:0}: Error finding container e0a8d61d638cefe03681236daa93a3db008be374cd4092c221b13a1dd13df387: Status 404 returned error can't find the container with id e0a8d61d638cefe03681236daa93a3db008be374cd4092c221b13a1dd13df387 Nov 02 06:50:17.913702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:17.913670 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" event={"ID":"b3c50225-9ac4-4438-a308-6daf361948b5","Type":"ContainerStarted","Data":"e0a8d61d638cefe03681236daa93a3db008be374cd4092c221b13a1dd13df387"} Nov 02 06:50:18.921697 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:18.921665 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" event={"ID":"b3c50225-9ac4-4438-a308-6daf361948b5","Type":"ContainerStarted","Data":"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb"} Nov 02 06:50:18.922151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:18.921767 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:19.033555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:19.033503 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" podStartSLOduration=2.15128135 podStartE2EDuration="3.033489547s" totalImagesPullingTime="882.208197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:19.030511231 +0000 UTC m=+1973.003845915" watchObservedRunningTime="2026-11-02 06:50:19.033489547 +0000 UTC m=+1973.006824230" Nov 02 06:50:21.990468 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:21.990429 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:50:21.990821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:21.990650 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" podUID="cf732e9e-6229-4df9-8081-97bd5222f66b" containerName="maas-api" containerID="cri-o://bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8" gracePeriod=30 Nov 02 06:50:22.252828 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.252723 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:50:22.299575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.299547 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\") pod \"cf732e9e-6229-4df9-8081-97bd5222f66b\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " Nov 02 06:50:22.299734 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.299660 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\" (UniqueName: \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\") pod \"cf732e9e-6229-4df9-8081-97bd5222f66b\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " Nov 02 06:50:22.299734 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.299711 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\") pod \"cf732e9e-6229-4df9-8081-97bd5222f66b\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " Nov 02 06:50:22.299935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.299919 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") pod \"cf732e9e-6229-4df9-8081-97bd5222f66b\" (UID: \"cf732e9e-6229-4df9-8081-97bd5222f66b\") " Nov 02 06:50:22.300125 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.300095 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca" pod "cf732e9e-6229-4df9-8081-97bd5222f66b" (UID: "cf732e9e-6229-4df9-8081-97bd5222f66b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:22.300618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.300589 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf732e9e-6229-4df9-8081-97bd5222f66b-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:22.302103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.302061 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls" pod "cf732e9e-6229-4df9-8081-97bd5222f66b" (UID: "cf732e9e-6229-4df9-8081-97bd5222f66b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:22.302214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.302142 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87" pod "cf732e9e-6229-4df9-8081-97bd5222f66b" (UID: "cf732e9e-6229-4df9-8081-97bd5222f66b"). InnerVolumeSpecName "kube-api-access-ddx87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:22.302214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.302178 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs" pod "cf732e9e-6229-4df9-8081-97bd5222f66b" (UID: "cf732e9e-6229-4df9-8081-97bd5222f66b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:22.401939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.401909 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:22.401939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.401938 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf732e9e-6229-4df9-8081-97bd5222f66b-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:22.402174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.401949 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddx87\" (UniqueName: \"kubernetes.io/projected/cf732e9e-6229-4df9-8081-97bd5222f66b-kube-api-access-ddx87\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:22.953537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.953503 6273 generic.go:356] "Generic (PLEG): container finished" podID="cf732e9e-6229-4df9-8081-97bd5222f66b" containerID="bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8" exitCode=0 Nov 02 06:50:22.953774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.953548 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" Nov 02 06:50:22.953774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.953587 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" event={"ID":"cf732e9e-6229-4df9-8081-97bd5222f66b","Type":"ContainerDied","Data":"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8"} Nov 02 06:50:22.953774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.953648 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb" event={"ID":"cf732e9e-6229-4df9-8081-97bd5222f66b","Type":"ContainerDied","Data":"9ea9cc027aa8a527eb9857baf0e735852b451e0c282b101736dfc274f2efd73f"} Nov 02 06:50:22.953774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.953673 6273 scope.go:122] "RemoveContainer" containerID="bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8" Nov 02 06:50:22.968587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.968558 6273 scope.go:122] "RemoveContainer" containerID="bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8" Nov 02 06:50:22.969032 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:22.968942 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8\": container with ID starting with bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8 not found: ID does not exist" containerID="bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8" Nov 02 06:50:22.969032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:22.968980 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8"} err="failed to get container status \"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8\": rpc error: code = NotFound desc = could not find container \"bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8\": container with ID starting with bc436f4de7fb0ee118604ebfe598c93ce159146d762aa50cea0f8202dc03eca8 not found: ID does not exist" Nov 02 06:50:23.122059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:23.121190 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:50:23.177245 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:23.177204 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-08bbf9-747548f467-x7grb"] Nov 02 06:50:24.181227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:24.181184 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf732e9e-6229-4df9-8081-97bd5222f66b" path="/var/lib/kubelet/pods/cf732e9e-6229-4df9-8081-97bd5222f66b/volumes" Nov 02 06:50:24.937347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:24.937318 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 06:50:25.926917 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.926879 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 06:50:25.927607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.927588 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cf732e9e-6229-4df9-8081-97bd5222f66b" containerName="maas-api" Nov 02 06:50:25.927607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.927606 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf732e9e-6229-4df9-8081-97bd5222f66b" containerName="maas-api" Nov 02 06:50:25.927794 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.927782 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf732e9e-6229-4df9-8081-97bd5222f66b" containerName="maas-api" Nov 02 06:50:25.933275 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.933251 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:25.985665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:25.985632 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 06:50:26.028086 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.028058 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8bc131-dockercfg-6m7kq\"" Nov 02 06:50:26.028284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.028059 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8bc131\"" Nov 02 06:50:26.065523 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.065486 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.065740 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.065641 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thrg\" (UniqueName: \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.096139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.096101 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 06:50:26.100790 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.100770 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.157032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.154575 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 06:50:26.167586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.167539 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thrg\" (UniqueName: \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.167756 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.167696 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.167756 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.167736 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vmx\" (UniqueName: \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.167884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.167771 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.229490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.217374 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8bc131\"" Nov 02 06:50:26.229490 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.223054 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.235709 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.235676 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thrg\" (UniqueName: \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\") pod \"payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.269555 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.269517 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.269729 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.269591 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vmx\" (UniqueName: \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.270448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.270425 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.296132 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.296109 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8bc131-dockercfg-6m7kq\"" Nov 02 06:50:26.299263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.299242 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:26.323361 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.323340 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vmx\" (UniqueName: \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\") pod \"payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.415259 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.415230 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.676003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.675960 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 06:50:26.682071 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:50:26.682045 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5834064e_dac1_4f44_8fdb_be8b4dfd9224.slice/crio-bf47786bd8a321ebba0eb40a684868621dc07e9b07be8600d18231af2efbe828 WatchSource:0}: Error finding container bf47786bd8a321ebba0eb40a684868621dc07e9b07be8600d18231af2efbe828: Status 404 returned error can't find the container with id bf47786bd8a321ebba0eb40a684868621dc07e9b07be8600d18231af2efbe828 Nov 02 06:50:26.804059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.804030 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 06:50:26.809052 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:50:26.809029 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736b71f9_b690_4d11_9c9f_5bf91999bf49.slice/crio-acd170f1a00a69e938a006d4616579d911795f90d691cbaa4a75b1731e8a2a84 WatchSource:0}: Error finding container acd170f1a00a69e938a006d4616579d911795f90d691cbaa4a75b1731e8a2a84: Status 404 returned error can't find the container with id acd170f1a00a69e938a006d4616579d911795f90d691cbaa4a75b1731e8a2a84 Nov 02 06:50:26.983420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.983335 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" event={"ID":"736b71f9-b690-4d11-9c9f-5bf91999bf49","Type":"ContainerStarted","Data":"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e"} Nov 02 06:50:26.983420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.983393 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:26.983420 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.983417 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" event={"ID":"736b71f9-b690-4d11-9c9f-5bf91999bf49","Type":"ContainerStarted","Data":"acd170f1a00a69e938a006d4616579d911795f90d691cbaa4a75b1731e8a2a84"} Nov 02 06:50:26.985080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.985053 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" event={"ID":"5834064e-dac1-4f44-8fdb-be8b4dfd9224","Type":"ContainerStarted","Data":"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f"} Nov 02 06:50:26.985192 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.985098 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" event={"ID":"5834064e-dac1-4f44-8fdb-be8b4dfd9224","Type":"ContainerStarted","Data":"bf47786bd8a321ebba0eb40a684868621dc07e9b07be8600d18231af2efbe828"} Nov 02 06:50:26.985192 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:26.985119 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:27.096994 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:27.096937 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" podStartSLOduration=1.096922963 podStartE2EDuration="1.096922963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:27.094416922 +0000 UTC m=+1981.067751641" watchObservedRunningTime="2026-11-02 06:50:27.096922963 +0000 UTC m=+1981.070257647" Nov 02 06:50:27.207854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:27.206251 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" podStartSLOduration=2.206231495 podStartE2EDuration="2.206231495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:27.202222833 +0000 UTC m=+1981.175557551" watchObservedRunningTime="2026-11-02 06:50:27.206231495 +0000 UTC m=+1981.179566180" Nov 02 06:50:30.979466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:30.979432 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:50:30.985451 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:30.985430 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.088168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.088138 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:50:31.131507 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131472 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpz8\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131523 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131710 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131662 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131717 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131747 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131831 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131812 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131848 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.131971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.131957 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.132043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.132006 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.233859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.233771 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.233894 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.233935 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234080 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234048 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpz8\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234119 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234261 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234397 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234495 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234473 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234559 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234538 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234585 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234633 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.234749 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.234540 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.235068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.235033 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.235341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.235322 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.236366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.236344 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.237119 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.237095 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.287421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.287392 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.287531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.287480 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpz8\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.302415 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.302392 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:31.733633 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.733596 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:50:31.740945 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:50:31.740914 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7636358f_0d9e_45c5_b571_ce4fd5cf80a1.slice/crio-272a460755c350fcd757a95013a811e1f222f85abef40d338cae759da1f5dd86 WatchSource:0}: Error finding container 272a460755c350fcd757a95013a811e1f222f85abef40d338cae759da1f5dd86: Status 404 returned error can't find the container with id 272a460755c350fcd757a95013a811e1f222f85abef40d338cae759da1f5dd86 Nov 02 06:50:31.743753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.743718 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:31.743832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.743790 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:31.743870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:31.743826 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:32.021649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.021558 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" event={"ID":"7636358f-0d9e-45c5-b571-ce4fd5cf80a1","Type":"ContainerStarted","Data":"c66576b3fe2d6f8a47bbda1c35291ed3b25d7b77d00e7eca7266083b9ee7eebf"} Nov 02 06:50:32.021649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.021624 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" event={"ID":"7636358f-0d9e-45c5-b571-ce4fd5cf80a1","Type":"ContainerStarted","Data":"272a460755c350fcd757a95013a811e1f222f85abef40d338cae759da1f5dd86"} Nov 02 06:50:32.138612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.138558 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podStartSLOduration=2.138543074 podStartE2EDuration="2.138543074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:32.135705648 +0000 UTC m=+1986.109040333" watchObservedRunningTime="2026-11-02 06:50:32.138543074 +0000 UTC m=+1986.111877814" Nov 02 06:50:32.302885 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.302779 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:32.304646 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.304610 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:32.304816 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:32.304679 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:33.302960 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:33.302919 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:33.303492 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:33.302979 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:34.302877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:34.302837 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:34.303074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:34.302893 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:35.303285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:35.303250 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:35.303702 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:35.303303 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:36.302774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:36.302737 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:36.302969 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:36.302794 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:37.303227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.303186 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 06:50:37.303627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.303253 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 06:50:37.783681 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.783645 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:50:37.887889 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.887849 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:50:37.995967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.995937 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 06:50:37.997226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:37.997205 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 06:50:38.402962 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:38.402931 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:38.403711 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:38.403690 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:38.404713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:38.404690 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:50:38.568969 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:38.568927 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:50:42.784577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:42.784534 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" podUID="71732abe-2d44-49a7-a716-630dddf9b1bb" containerName="payload-processing" containerID="cri-o://b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd" gracePeriod=25 Nov 02 06:50:42.888649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:42.888605 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" podUID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" containerName="payload-pre-processing" containerID="cri-o://981f6719d3827db0aa9df095de347e3b1773c070f54dc29bf79bea162679312f" gracePeriod=25 Nov 02 06:50:43.078359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.077942 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:50:43.122320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.122282 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\") pod \"71732abe-2d44-49a7-a716-630dddf9b1bb\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " Nov 02 06:50:43.122515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.122432 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\" (UniqueName: \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\") pod \"71732abe-2d44-49a7-a716-630dddf9b1bb\" (UID: \"71732abe-2d44-49a7-a716-630dddf9b1bb\") " Nov 02 06:50:43.122920 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.122869 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume" pod "71732abe-2d44-49a7-a716-630dddf9b1bb" (UID: "71732abe-2d44-49a7-a716-630dddf9b1bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:43.123279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.123253 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71732abe-2d44-49a7-a716-630dddf9b1bb-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:43.124712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.124684 6273 generic.go:356] "Generic (PLEG): container finished" podID="71732abe-2d44-49a7-a716-630dddf9b1bb" containerID="b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd" exitCode=0 Nov 02 06:50:43.124856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.124750 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" Nov 02 06:50:43.124856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.124827 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" event={"ID":"71732abe-2d44-49a7-a716-630dddf9b1bb","Type":"ContainerDied","Data":"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd"} Nov 02 06:50:43.125014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.124877 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75" event={"ID":"71732abe-2d44-49a7-a716-630dddf9b1bb","Type":"ContainerDied","Data":"ec09f0afc941a34fa81acf21376aa946ae6f068150cff139f79f97da33d23dcd"} Nov 02 06:50:43.125014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.124955 6273 scope.go:122] "RemoveContainer" containerID="b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd" Nov 02 06:50:43.125653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.125626 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q" pod "71732abe-2d44-49a7-a716-630dddf9b1bb" (UID: "71732abe-2d44-49a7-a716-630dddf9b1bb"). InnerVolumeSpecName "kube-api-access-smv8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:43.127366 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.127340 6273 generic.go:356] "Generic (PLEG): container finished" podID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" containerID="981f6719d3827db0aa9df095de347e3b1773c070f54dc29bf79bea162679312f" exitCode=0 Nov 02 06:50:43.127506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.127479 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" event={"ID":"b3a78f19-1e8c-4433-a6cf-610311ad4c7d","Type":"ContainerDied","Data":"981f6719d3827db0aa9df095de347e3b1773c070f54dc29bf79bea162679312f"} Nov 02 06:50:43.149824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.149799 6273 scope.go:122] "RemoveContainer" containerID="b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd" Nov 02 06:50:43.150250 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:43.150220 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd\": container with ID starting with b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd not found: ID does not exist" containerID="b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd" Nov 02 06:50:43.150362 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.150262 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd"} err="failed to get container status \"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd\": rpc error: code = NotFound desc = could not find container \"b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd\": container with ID starting with b774384a12542ffe1c9a800b8813e958869ccdbac0cf331ee10c9b0f439231dd not found: ID does not exist" Nov 02 06:50:43.163155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.163131 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:50:43.224572 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.224537 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\") pod \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " Nov 02 06:50:43.224796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.224735 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\" (UniqueName: \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\") pod \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\" (UID: \"b3a78f19-1e8c-4433-a6cf-610311ad4c7d\") " Nov 02 06:50:43.225039 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.225013 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume" pod "b3a78f19-1e8c-4433-a6cf-610311ad4c7d" (UID: "b3a78f19-1e8c-4433-a6cf-610311ad4c7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:43.225494 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.225470 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smv8q\" (UniqueName: \"kubernetes.io/projected/71732abe-2d44-49a7-a716-630dddf9b1bb-kube-api-access-smv8q\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:43.225653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.225628 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:43.227156 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.227127 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt" pod "b3a78f19-1e8c-4433-a6cf-610311ad4c7d" (UID: "b3a78f19-1e8c-4433-a6cf-610311ad4c7d"). InnerVolumeSpecName "kube-api-access-pmjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:43.326972 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.326893 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjvt\" (UniqueName: \"kubernetes.io/projected/b3a78f19-1e8c-4433-a6cf-610311ad4c7d-kube-api-access-pmjvt\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:43.539810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.539775 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:50:43.543678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:43.543646 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29abe7-6598b84467-kcg75"] Nov 02 06:50:44.135765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.135679 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" Nov 02 06:50:44.135765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.135725 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn" event={"ID":"b3a78f19-1e8c-4433-a6cf-610311ad4c7d","Type":"ContainerDied","Data":"0b47c40305fb24e4e7bc07543b1f70496a02230c30a476e454689b4875bb6cd4"} Nov 02 06:50:44.136292 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.135791 6273 scope.go:122] "RemoveContainer" containerID="981f6719d3827db0aa9df095de347e3b1773c070f54dc29bf79bea162679312f" Nov 02 06:50:44.179888 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.179860 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71732abe-2d44-49a7-a716-630dddf9b1bb" path="/var/lib/kubelet/pods/71732abe-2d44-49a7-a716-630dddf9b1bb/volumes" Nov 02 06:50:44.290360 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.290331 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:50:44.295736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.295711 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29abe7-7c65c976f5-bbwvn"] Nov 02 06:50:44.478847 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:44.478816 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:50:46.180714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:46.180682 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" path="/var/lib/kubelet/pods/b3a78f19-1e8c-4433-a6cf-610311ad4c7d/volumes" Nov 02 06:50:49.479113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.479073 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" podUID="992a7513-01ed-4312-8920-138ee72d3e38" containerName="payload-pre-processing" containerID="cri-o://7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493" gracePeriod=25 Nov 02 06:50:49.540117 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:49.540090 6273 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992a7513_01ed_4312_8920_138ee72d3e38.slice/crio-7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493.scope\": RecentStats: unable to find data in memory cache]" Nov 02 06:50:49.761504 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.761480 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:50:49.913405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.913377 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\") pod \"992a7513-01ed-4312-8920-138ee72d3e38\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " Nov 02 06:50:49.913599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.913462 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\" (UniqueName: \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\") pod \"992a7513-01ed-4312-8920-138ee72d3e38\" (UID: \"992a7513-01ed-4312-8920-138ee72d3e38\") " Nov 02 06:50:49.913791 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.913766 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume" pod "992a7513-01ed-4312-8920-138ee72d3e38" (UID: "992a7513-01ed-4312-8920-138ee72d3e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:49.914130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.914107 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992a7513-01ed-4312-8920-138ee72d3e38-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:49.915545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:49.915521 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh" pod "992a7513-01ed-4312-8920-138ee72d3e38" (UID: "992a7513-01ed-4312-8920-138ee72d3e38"). InnerVolumeSpecName "kube-api-access-hzcmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:50.015305 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.015243 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcmh\" (UniqueName: \"kubernetes.io/projected/992a7513-01ed-4312-8920-138ee72d3e38-kube-api-access-hzcmh\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:50.183218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.183185 6273 generic.go:356] "Generic (PLEG): container finished" podID="992a7513-01ed-4312-8920-138ee72d3e38" containerID="7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493" exitCode=0 Nov 02 06:50:50.183371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.183236 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" Nov 02 06:50:50.183371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.183257 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" event={"ID":"992a7513-01ed-4312-8920-138ee72d3e38","Type":"ContainerDied","Data":"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493"} Nov 02 06:50:50.183371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.183308 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59" event={"ID":"992a7513-01ed-4312-8920-138ee72d3e38","Type":"ContainerDied","Data":"bf94151a4bd26590ecb277c7c88ae72def12dfca3bf4eca421f5e204bbbffc67"} Nov 02 06:50:50.183371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.183329 6273 scope.go:122] "RemoveContainer" containerID="7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493" Nov 02 06:50:50.197103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.195916 6273 scope.go:122] "RemoveContainer" containerID="7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493" Nov 02 06:50:50.197103 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:50:50.196561 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493\": container with ID starting with 7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493 not found: ID does not exist" containerID="7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493" Nov 02 06:50:50.197103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.196599 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493"} err="failed to get container status \"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493\": rpc error: code = NotFound desc = could not find container \"7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493\": container with ID starting with 7c9e07cbd3081bdec9afe2edbad9623bd91b11f982558be579fcf9484accc493 not found: ID does not exist" Nov 02 06:50:50.338204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.338094 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:50:50.384895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:50.384864 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d13c3-5dcd66975-hll59"] Nov 02 06:50:51.132388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:51.132353 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:50:52.180756 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:52.180721 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992a7513-01ed-4312-8920-138ee72d3e38" path="/var/lib/kubelet/pods/992a7513-01ed-4312-8920-138ee72d3e38/volumes" Nov 02 06:50:57.142827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.142783 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-c7b52e"] Nov 02 06:50:57.143658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143636 6273 cpu_manager.go:414] "containerMap: removing container" podUID="992a7513-01ed-4312-8920-138ee72d3e38" containerName="payload-pre-processing" Nov 02 06:50:57.143763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143660 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992a7513-01ed-4312-8920-138ee72d3e38" containerName="payload-pre-processing" Nov 02 06:50:57.143763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143702 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" containerName="payload-pre-processing" Nov 02 06:50:57.143763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143712 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" containerName="payload-pre-processing" Nov 02 06:50:57.143763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143760 6273 cpu_manager.go:414] "containerMap: removing container" podUID="71732abe-2d44-49a7-a716-630dddf9b1bb" containerName="payload-processing" Nov 02 06:50:57.144031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143768 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71732abe-2d44-49a7-a716-630dddf9b1bb" containerName="payload-processing" Nov 02 06:50:57.144031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143917 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="992a7513-01ed-4312-8920-138ee72d3e38" containerName="payload-pre-processing" Nov 02 06:50:57.144031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143938 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a78f19-1e8c-4433-a6cf-610311ad4c7d" containerName="payload-pre-processing" Nov 02 06:50:57.144031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.143956 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="71732abe-2d44-49a7-a716-630dddf9b1bb" containerName="payload-processing" Nov 02 06:50:57.148865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.148843 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:50:57.199405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.199374 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-c7b52e"] Nov 02 06:50:57.241977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.241950 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g9gq\"" Nov 02 06:50:57.311932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.311892 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfm84\" (UniqueName: \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\") pod \"test-curl-7135-c7b52e\" (UID: \"ced5afde-22eb-4b67-93a0-5e4db9406848\") " pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:50:57.412618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.412586 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfm84\" (UniqueName: \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\") pod \"test-curl-7135-c7b52e\" (UID: \"ced5afde-22eb-4b67-93a0-5e4db9406848\") " pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:50:57.468772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.468739 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfm84\" (UniqueName: \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\") pod \"test-curl-7135-c7b52e\" (UID: \"ced5afde-22eb-4b67-93a0-5e4db9406848\") " pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:50:57.760981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:57.760898 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:50:58.133290 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:58.133215 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-c7b52e"] Nov 02 06:50:58.139260 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:50:58.139232 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podced5afde_22eb_4b67_93a0_5e4db9406848.slice/crio-f5911529a82a901b895d9a381125aa81d58bf41dcf8f0d958d289447fef186b2 WatchSource:0}: Error finding container f5911529a82a901b895d9a381125aa81d58bf41dcf8f0d958d289447fef186b2: Status 404 returned error can't find the container with id f5911529a82a901b895d9a381125aa81d58bf41dcf8f0d958d289447fef186b2 Nov 02 06:50:58.235878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:50:58.235846 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-c7b52e" event={"ID":"ced5afde-22eb-4b67-93a0-5e4db9406848","Type":"ContainerStarted","Data":"f5911529a82a901b895d9a381125aa81d58bf41dcf8f0d958d289447fef186b2"} Nov 02 06:51:00.249833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:00.249798 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-c7b52e" event={"ID":"ced5afde-22eb-4b67-93a0-5e4db9406848","Type":"ContainerStarted","Data":"2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4"} Nov 02 06:51:00.357043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:00.356969 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-c7b52e" podStartSLOduration=2.161732311 podStartE2EDuration="3.356955858s" totalImagesPullingTime="1.195223547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:51:00.355573508 +0000 UTC m=+2014.328908193" watchObservedRunningTime="2026-11-02 06:51:00.356955858 +0000 UTC m=+2014.330290541" Nov 02 06:51:01.750853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:01.750821 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-c7b52e"] Nov 02 06:51:01.755378 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:01.755347 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-c7b52e"] Nov 02 06:51:02.262429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:02.262387 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-c7b52e" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" containerName="test-curl-7135-c7b52e" containerID="cri-o://2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4" gracePeriod=2 Nov 02 06:51:04.519471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:04.519409 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:51:04.603369 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:04.603325 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\" (UniqueName: \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\") pod \"ced5afde-22eb-4b67-93a0-5e4db9406848\" (UID: \"ced5afde-22eb-4b67-93a0-5e4db9406848\") " Nov 02 06:51:04.605724 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:04.605690 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84" pod "ced5afde-22eb-4b67-93a0-5e4db9406848" (UID: "ced5afde-22eb-4b67-93a0-5e4db9406848"). InnerVolumeSpecName "kube-api-access-dfm84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:04.705272 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:04.705242 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfm84\" (UniqueName: \"kubernetes.io/projected/ced5afde-22eb-4b67-93a0-5e4db9406848-kube-api-access-dfm84\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:05.283767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:05.283729 6273 generic.go:356] "Generic (PLEG): container finished" podID="ced5afde-22eb-4b67-93a0-5e4db9406848" containerID="2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4" exitCode=137 Nov 02 06:51:05.283951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:05.283782 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:51:05.283951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:05.283793 6273 scope.go:122] "RemoveContainer" containerID="2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4" Nov 02 06:51:05.293787 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:05.293771 6273 scope.go:122] "RemoveContainer" containerID="2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4" Nov 02 06:51:05.294063 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:05.294045 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4\": container with ID starting with 2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4 not found: ID does not exist" containerID="2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4" Nov 02 06:51:05.294118 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:05.294070 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4"} err="failed to get container status \"2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4\": rpc error: code = NotFound desc = could not find container \"2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4\": container with ID starting with 2e9bc6589c7ad5ee73ef04ea1981c35a5156622cb2a9efdc18436e1f40e0e1a4 not found: ID does not exist" Nov 02 06:51:05.340363 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:05.340336 6273 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-c7b52e\" is forbidden: User \"system:node:ip-10-0-142-103.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-103.us-west-2.compute.internal' and this object" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:51:06.182155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:06.182125 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" path="/var/lib/kubelet/pods/ced5afde-22eb-4b67-93a0-5e4db9406848/volumes" Nov 02 06:51:06.223846 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:06.223809 6273 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-c7b52e\" is forbidden: User \"system:node:ip-10-0-142-103.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-103.us-west-2.compute.internal' and this object" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" pod="opendatahub/test-curl-7135-c7b52e" Nov 02 06:51:08.560362 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:08.560319 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:51:08.720025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:08.719976 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:51:08.720227 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:08.720189 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" containerID="cri-o://880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0" gracePeriod=30 Nov 02 06:51:10.028029 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:10.027967 6273 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:51:10.028402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:10.028068 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:51:14.006144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.006064 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:51:14.132810 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.132778 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.132898 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.132962 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133082 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133100 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:14.133155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133122 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133154 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133241 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133240 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:14.133327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133278 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133309 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\") pod \"7b9033df-a130-4d5a-8037-b416d78c4e19\" (UID: \"7b9033df-a130-4d5a-8037-b416d78c4e19\") " Nov 02 06:51:14.133848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133819 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:14.134002 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133894 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.134002 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133917 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.134141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.133969 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:14.134141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.134011 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:14.135478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.135447 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:14.135600 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.135585 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "kube-api-access-q9ffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:14.136040 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.135973 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:14.136285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.136260 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo" pod "7b9033df-a130-4d5a-8037-b416d78c4e19" (UID: "7b9033df-a130-4d5a-8037-b416d78c4e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:14.235570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235531 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b9033df-a130-4d5a-8037-b416d78c4e19-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235563 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235573 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235582 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235591 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ffw\" (UniqueName: \"kubernetes.io/projected/7b9033df-a130-4d5a-8037-b416d78c4e19-kube-api-access-q9ffw\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235600 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b9033df-a130-4d5a-8037-b416d78c4e19-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.235808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.235610 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b9033df-a130-4d5a-8037-b416d78c4e19-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:14.348716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.348631 6273 generic.go:356] "Generic (PLEG): container finished" podID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerID="880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0" exitCode=0 Nov 02 06:51:14.348716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.348699 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" Nov 02 06:51:14.348917 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.348719 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" event={"ID":"7b9033df-a130-4d5a-8037-b416d78c4e19","Type":"ContainerDied","Data":"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0"} Nov 02 06:51:14.348917 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.348789 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6" event={"ID":"7b9033df-a130-4d5a-8037-b416d78c4e19","Type":"ContainerDied","Data":"abd39bc6bb926dceaa988c7d080906de9c85f5058eb3d75dcb1a47601e26c9ce"} Nov 02 06:51:14.348917 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.348821 6273 scope.go:122] "RemoveContainer" containerID="880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0" Nov 02 06:51:14.359128 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.359110 6273 scope.go:122] "RemoveContainer" containerID="880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0" Nov 02 06:51:14.359393 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:14.359374 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0\": container with ID starting with 880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0 not found: ID does not exist" containerID="880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0" Nov 02 06:51:14.359442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.359402 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0"} err="failed to get container status \"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0\": rpc error: code = NotFound desc = could not find container \"880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0\": container with ID starting with 880974b370698d3537ff55cd794b85cfc167006b5add7ebc3373cd8ec53f93e0 not found: ID does not exist" Nov 02 06:51:14.504127 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.504091 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:51:14.510524 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:14.510485 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-nhlt6"] Nov 02 06:51:16.183864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:16.183827 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" path="/var/lib/kubelet/pods/7b9033df-a130-4d5a-8037-b416d78c4e19/volumes" Nov 02 06:51:27.300418 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.300382 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:51:27.300914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.300603 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" podUID="41482f2e-646a-421c-81f8-0c597ccbc412" containerName="maas-api" containerID="cri-o://01c4c7079d113aaeae382b9d03cfbe9219dc0eb18bec0ca43f3aaa31d4f99aed" gracePeriod=30 Nov 02 06:51:27.438001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.437955 6273 generic.go:356] "Generic (PLEG): container finished" podID="41482f2e-646a-421c-81f8-0c597ccbc412" containerID="01c4c7079d113aaeae382b9d03cfbe9219dc0eb18bec0ca43f3aaa31d4f99aed" exitCode=0 Nov 02 06:51:27.438123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.438064 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" event={"ID":"41482f2e-646a-421c-81f8-0c597ccbc412","Type":"ContainerDied","Data":"01c4c7079d113aaeae382b9d03cfbe9219dc0eb18bec0ca43f3aaa31d4f99aed"} Nov 02 06:51:27.588003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.587962 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:51:27.716431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.716260 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\") pod \"41482f2e-646a-421c-81f8-0c597ccbc412\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " Nov 02 06:51:27.716431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.716366 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") pod \"41482f2e-646a-421c-81f8-0c597ccbc412\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " Nov 02 06:51:27.716669 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.716618 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\") pod \"41482f2e-646a-421c-81f8-0c597ccbc412\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " Nov 02 06:51:27.716717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.716702 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\" (UniqueName: \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\") pod \"41482f2e-646a-421c-81f8-0c597ccbc412\" (UID: \"41482f2e-646a-421c-81f8-0c597ccbc412\") " Nov 02 06:51:27.716784 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.716749 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca" pod "41482f2e-646a-421c-81f8-0c597ccbc412" (UID: "41482f2e-646a-421c-81f8-0c597ccbc412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:27.718351 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.718322 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41482f2e-646a-421c-81f8-0c597ccbc412-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:27.723003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.719791 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls" pod "41482f2e-646a-421c-81f8-0c597ccbc412" (UID: "41482f2e-646a-421c-81f8-0c597ccbc412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:27.723003 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.719896 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs" pod "41482f2e-646a-421c-81f8-0c597ccbc412" (UID: "41482f2e-646a-421c-81f8-0c597ccbc412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:27.723407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.723381 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw" pod "41482f2e-646a-421c-81f8-0c597ccbc412" (UID: "41482f2e-646a-421c-81f8-0c597ccbc412"). InnerVolumeSpecName "kube-api-access-d7fzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:27.819500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.819464 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:27.819500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.819498 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41482f2e-646a-421c-81f8-0c597ccbc412-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:27.819704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:27.819511 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fzw\" (UniqueName: \"kubernetes.io/projected/41482f2e-646a-421c-81f8-0c597ccbc412-kube-api-access-d7fzw\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:28.445485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:28.445401 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" Nov 02 06:51:28.445485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:28.445428 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm" event={"ID":"41482f2e-646a-421c-81f8-0c597ccbc412","Type":"ContainerDied","Data":"71130319ed593510bf1aec298f325ffff2c7e32bd2a667175792a49db03c2e57"} Nov 02 06:51:28.446065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:28.445493 6273 scope.go:122] "RemoveContainer" containerID="01c4c7079d113aaeae382b9d03cfbe9219dc0eb18bec0ca43f3aaa31d4f99aed" Nov 02 06:51:28.605510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:28.605477 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:51:28.651120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:28.651082 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8ecb23-6785656864-xbthm"] Nov 02 06:51:30.180315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:30.180280 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41482f2e-646a-421c-81f8-0c597ccbc412" path="/var/lib/kubelet/pods/41482f2e-646a-421c-81f8-0c597ccbc412/volumes" Nov 02 06:51:38.553153 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:38.553114 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:51:43.674999 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.674951 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 06:51:43.675666 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675648 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" Nov 02 06:51:43.675712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675668 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" Nov 02 06:51:43.675712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675683 6273 cpu_manager.go:414] "containerMap: removing container" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" containerName="test-curl-7135-c7b52e" Nov 02 06:51:43.675712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675688 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" containerName="test-curl-7135-c7b52e" Nov 02 06:51:43.675712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675704 6273 cpu_manager.go:414] "containerMap: removing container" podUID="41482f2e-646a-421c-81f8-0c597ccbc412" containerName="maas-api" Nov 02 06:51:43.675712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675709 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41482f2e-646a-421c-81f8-0c597ccbc412" containerName="maas-api" Nov 02 06:51:43.675870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675814 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9033df-a130-4d5a-8037-b416d78c4e19" containerName="istio-proxy" Nov 02 06:51:43.675870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675824 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced5afde-22eb-4b67-93a0-5e4db9406848" containerName="test-curl-7135-c7b52e" Nov 02 06:51:43.675870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.675833 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="41482f2e-646a-421c-81f8-0c597ccbc412" containerName="maas-api" Nov 02 06:51:43.681000 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.680971 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.707551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.707526 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.707664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.707577 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.707735 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.707710 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.707819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.707804 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.730918 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.730896 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8bc131-serving-cert\"" Nov 02 06:51:43.780699 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.780673 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 06:51:43.809740 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.809711 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.809889 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.809795 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.809889 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.809852 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.810058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.809928 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.810129 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:43.810089 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-serving-cert: secret "maas-api-e2e-inf-a-8bc131-serving-cert" not found Nov 02 06:51:43.810191 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:43.810150 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls podName:b9bbee33-c01f-4d8e-8664-bb25a4534f44 nodeName:}" failed. No retries permitted until 2026-11-02 06:51:44.310130221 +0000 UTC m=+2058.283464903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls") pod "maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" (UID: "b9bbee33-c01f-4d8e-8664-bb25a4534f44") : secret "maas-api-e2e-inf-a-8bc131-serving-cert" not found Nov 02 06:51:43.810550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.810521 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.812534 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.812507 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:43.866381 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:43.866359 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:44.316575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:44.316538 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:44.318978 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:44.318951 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") pod \"maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:44.595130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:44.595038 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:44.983547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:44.983514 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 06:51:44.989801 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:51:44.989770 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bbee33_c01f_4d8e_8664_bb25a4534f44.slice/crio-8a4b508a81705451c7d69f606ea18cc61c8a6c607b08721eb721b84eb5f0221f WatchSource:0}: Error finding container 8a4b508a81705451c7d69f606ea18cc61c8a6c607b08721eb721b84eb5f0221f: Status 404 returned error can't find the container with id 8a4b508a81705451c7d69f606ea18cc61c8a6c607b08721eb721b84eb5f0221f Nov 02 06:51:45.555271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:45.555238 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" event={"ID":"b9bbee33-c01f-4d8e-8664-bb25a4534f44","Type":"ContainerStarted","Data":"8a4b508a81705451c7d69f606ea18cc61c8a6c607b08721eb721b84eb5f0221f"} Nov 02 06:51:46.565349 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:46.565309 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" event={"ID":"b9bbee33-c01f-4d8e-8664-bb25a4534f44","Type":"ContainerStarted","Data":"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327"} Nov 02 06:51:46.565951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:46.565931 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:46.678516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:46.678460 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" podStartSLOduration=2.686230619 podStartE2EDuration="3.678445349s" totalImagesPullingTime="992.21473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:51:46.676433076 +0000 UTC m=+2060.649767774" watchObservedRunningTime="2026-11-02 06:51:46.678445349 +0000 UTC m=+2060.651780033" Nov 02 06:51:49.821537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:49.821504 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:51:52.580714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:52.580681 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 06:51:53.553024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.552959 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:51:53.553268 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.553220 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" podUID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" containerName="maas-api" containerID="cri-o://0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd" gracePeriod=30 Nov 02 06:51:53.836019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.835977 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:51:53.942357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.942323 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") pod \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " Nov 02 06:51:53.942569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.942401 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\") pod \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " Nov 02 06:51:53.942569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.942451 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\" (UniqueName: \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\") pod \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " Nov 02 06:51:53.942569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.942537 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\") pod \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\" (UID: \"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3\") " Nov 02 06:51:53.943252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.943202 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca" pod "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" (UID: "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:53.946545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.946139 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls" pod "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" (UID: "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:53.946545 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.946180 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85" pod "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" (UID: "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3"). InnerVolumeSpecName "kube-api-access-sjn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:53.946854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:53.946817 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs" pod "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" (UID: "506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:54.044827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.044785 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:54.044827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.044816 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:54.044827 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.044828 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjn85\" (UniqueName: \"kubernetes.io/projected/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-kube-api-access-sjn85\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:54.045088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.044843 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:54.629066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.628260 6273 generic.go:356] "Generic (PLEG): container finished" podID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" containerID="0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd" exitCode=0 Nov 02 06:51:54.629066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.628347 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" Nov 02 06:51:54.629066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.628377 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" event={"ID":"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3","Type":"ContainerDied","Data":"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd"} Nov 02 06:51:54.629066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.628432 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs" event={"ID":"506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3","Type":"ContainerDied","Data":"074a81b70f6f9c740dcb95cba720b785ded1cf6fb1d3a51081537de54f0df301"} Nov 02 06:51:54.629066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.628463 6273 scope.go:122] "RemoveContainer" containerID="0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd" Nov 02 06:51:54.650422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.650372 6273 scope.go:122] "RemoveContainer" containerID="0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd" Nov 02 06:51:54.651496 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:54.650960 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd\": container with ID starting with 0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd not found: ID does not exist" containerID="0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd" Nov 02 06:51:54.651496 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.651035 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd"} err="failed to get container status \"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd\": rpc error: code = NotFound desc = could not find container \"0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd\": container with ID starting with 0c90c2529c3ee6c24daa95e70d4ebe2be29fea072fd7f9424d3deafe1a4d58fd not found: ID does not exist" Nov 02 06:51:54.784452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.784419 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:51:54.821976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.821932 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" podUID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" containerName="payload-processing" containerID="cri-o://4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b" gracePeriod=25 Nov 02 06:51:54.831379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.831353 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-29abe7-545f95484c-z9mfs"] Nov 02 06:51:54.833731 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.833711 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:51:54.833927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:54.833889 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" podUID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" containerName="maas-api" containerID="cri-o://a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d" gracePeriod=30 Nov 02 06:51:55.118639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.118614 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:51:55.160319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.160286 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\" (UniqueName: \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\") pod \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " Nov 02 06:51:55.160535 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.160517 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\") pod \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\" (UID: \"d6a1ec1a-144d-4116-9b04-afcb89c3ed00\") " Nov 02 06:51:55.160927 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.160901 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume" pod "d6a1ec1a-144d-4116-9b04-afcb89c3ed00" (UID: "d6a1ec1a-144d-4116-9b04-afcb89c3ed00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:55.161179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.161161 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.162409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.162382 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x" pod "d6a1ec1a-144d-4116-9b04-afcb89c3ed00" (UID: "d6a1ec1a-144d-4116-9b04-afcb89c3ed00"). InnerVolumeSpecName "kube-api-access-6p86x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:55.166644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.166591 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:51:55.262542 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.262510 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\") pod \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " Nov 02 06:51:55.262730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.262569 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\") pod \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " Nov 02 06:51:55.262730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.262639 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\" (UniqueName: \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\") pod \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " Nov 02 06:51:55.262730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.262710 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") pod \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\" (UID: \"dfecce48-759e-4b7d-85bc-d834ec21d0a6\") " Nov 02 06:51:55.262981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.262943 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca" pod "dfecce48-759e-4b7d-85bc-d834ec21d0a6" (UID: "dfecce48-759e-4b7d-85bc-d834ec21d0a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:55.263319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.263252 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfecce48-759e-4b7d-85bc-d834ec21d0a6-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.263319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.263288 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p86x\" (UniqueName: \"kubernetes.io/projected/d6a1ec1a-144d-4116-9b04-afcb89c3ed00-kube-api-access-6p86x\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.264774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.264739 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs" pod "dfecce48-759e-4b7d-85bc-d834ec21d0a6" (UID: "dfecce48-759e-4b7d-85bc-d834ec21d0a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:55.264982 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.264960 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls" pod "dfecce48-759e-4b7d-85bc-d834ec21d0a6" (UID: "dfecce48-759e-4b7d-85bc-d834ec21d0a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:55.265347 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.265330 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4" pod "dfecce48-759e-4b7d-85bc-d834ec21d0a6" (UID: "dfecce48-759e-4b7d-85bc-d834ec21d0a6"). InnerVolumeSpecName "kube-api-access-bpkg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:55.364452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.364424 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpkg4\" (UniqueName: \"kubernetes.io/projected/dfecce48-759e-4b7d-85bc-d834ec21d0a6-kube-api-access-bpkg4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.364452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.364450 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.364452 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.364459 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfecce48-759e-4b7d-85bc-d834ec21d0a6-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.636287 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.636195 6273 generic.go:356] "Generic (PLEG): container finished" podID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" containerID="4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b" exitCode=0 Nov 02 06:51:55.636287 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.636236 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" event={"ID":"d6a1ec1a-144d-4116-9b04-afcb89c3ed00","Type":"ContainerDied","Data":"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b"} Nov 02 06:51:55.636287 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.636253 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" Nov 02 06:51:55.636540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.636296 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c" event={"ID":"d6a1ec1a-144d-4116-9b04-afcb89c3ed00","Type":"ContainerDied","Data":"c573a478f6c29b237713dc9e7e99f700c29ada2ebdd0bbfc0744f40dff8f3bec"} Nov 02 06:51:55.636540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.636322 6273 scope.go:122] "RemoveContainer" containerID="4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b" Nov 02 06:51:55.639292 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.639267 6273 generic.go:356] "Generic (PLEG): container finished" podID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" containerID="a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d" exitCode=0 Nov 02 06:51:55.639431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.639298 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" event={"ID":"dfecce48-759e-4b7d-85bc-d834ec21d0a6","Type":"ContainerDied","Data":"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d"} Nov 02 06:51:55.639431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.639322 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" Nov 02 06:51:55.639431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.639346 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s" event={"ID":"dfecce48-759e-4b7d-85bc-d834ec21d0a6","Type":"ContainerDied","Data":"ec857e51285f6ad3c8d5727028ec817708c217198b44549ae4934563acc98e30"} Nov 02 06:51:55.649350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.649323 6273 scope.go:122] "RemoveContainer" containerID="4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b" Nov 02 06:51:55.649659 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:55.649622 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b\": container with ID starting with 4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b not found: ID does not exist" containerID="4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b" Nov 02 06:51:55.649741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.649663 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b"} err="failed to get container status \"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b\": rpc error: code = NotFound desc = could not find container \"4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b\": container with ID starting with 4985a5ae6e7049c6bc34fd59228d1b5b5ddc2f2e5df51f7e3a59aee8b631cd9b not found: ID does not exist" Nov 02 06:51:55.649741 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.649685 6273 scope.go:122] "RemoveContainer" containerID="a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d" Nov 02 06:51:55.658659 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.658634 6273 scope.go:122] "RemoveContainer" containerID="a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d" Nov 02 06:51:55.658928 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:51:55.658907 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d\": container with ID starting with a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d not found: ID does not exist" containerID="a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d" Nov 02 06:51:55.659023 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.658935 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d"} err="failed to get container status \"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d\": rpc error: code = NotFound desc = could not find container \"a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d\": container with ID starting with a5de7fdc3c3c51e0c05b2876666ef85d554aa6956421839e76dace0c0a67933d not found: ID does not exist" Nov 02 06:51:55.792974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.792908 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:51:55.795864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.795841 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b53e-699fb78fff-smw4c"] Nov 02 06:51:55.898584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.898513 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:51:55.900951 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:55.900927 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d13c3-7494f76b55-vg49s"] Nov 02 06:51:56.179923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:56.179891 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" path="/var/lib/kubelet/pods/506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3/volumes" Nov 02 06:51:56.180323 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:56.180313 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" path="/var/lib/kubelet/pods/d6a1ec1a-144d-4116-9b04-afcb89c3ed00/volumes" Nov 02 06:51:56.180720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:56.180705 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" path="/var/lib/kubelet/pods/dfecce48-759e-4b7d-85bc-d834ec21d0a6/volumes" Nov 02 06:51:57.341425 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:57.341392 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:51:58.230687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.230650 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 06:51:58.231312 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231296 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" containerName="payload-processing" Nov 02 06:51:58.231371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231314 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" containerName="payload-processing" Nov 02 06:51:58.231371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231339 6273 cpu_manager.go:414] "containerMap: removing container" podUID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" containerName="maas-api" Nov 02 06:51:58.231371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231345 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" containerName="maas-api" Nov 02 06:51:58.231475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231377 6273 cpu_manager.go:414] "containerMap: removing container" podUID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" containerName="maas-api" Nov 02 06:51:58.231475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231382 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" containerName="maas-api" Nov 02 06:51:58.231475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231464 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfecce48-759e-4b7d-85bc-d834ec21d0a6" containerName="maas-api" Nov 02 06:51:58.231475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231473 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a1ec1a-144d-4116-9b04-afcb89c3ed00" containerName="payload-processing" Nov 02 06:51:58.231604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.231482 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="506bf7ec-f8f8-42a6-9fb3-b5cd9bcef4d3" containerName="maas-api" Nov 02 06:51:58.236576 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.236550 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.290450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.290421 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 06:51:58.311401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.311356 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7km\" (UniqueName: \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.311601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.311468 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.335096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.333830 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5a85b0-dockercfg-d7fvd\"" Nov 02 06:51:58.335096 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.334217 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a85b0\"" Nov 02 06:51:58.396161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.396133 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 06:51:58.401971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.401945 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.413175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.413148 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7km\" (UniqueName: \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.413313 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.413219 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.413894 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.413873 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.455513 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.455489 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 06:51:58.469207 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.469180 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7km\" (UniqueName: \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\") pod \"payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.518137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.518040 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbnl\" (UniqueName: \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.518137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.518101 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.550881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.550850 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:58.620473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.620436 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbnl\" (UniqueName: \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.620612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.620496 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.621562 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.621533 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.675751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.675720 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbnl\" (UniqueName: \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\") pod \"payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.724273 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.724244 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:58.928509 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:58.928478 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 06:51:58.934737 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:51:58.934711 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d438b90_6711_4ce0_bb13_5c3684e8b4b2.slice/crio-e74719556b95dd0b4ffe4d46b29159c23f4f6cc34942e858a32ea26168cce639 WatchSource:0}: Error finding container e74719556b95dd0b4ffe4d46b29159c23f4f6cc34942e858a32ea26168cce639: Status 404 returned error can't find the container with id e74719556b95dd0b4ffe4d46b29159c23f4f6cc34942e858a32ea26168cce639 Nov 02 06:51:59.099038 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.098951 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 06:51:59.104857 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:51:59.104831 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2dd1a73_8743_4298_b812_eb823cede79d.slice/crio-efe9605abd9125d93862e36dcd746fd86f898a031fd8fd143e32a266a8d8f87b WatchSource:0}: Error finding container efe9605abd9125d93862e36dcd746fd86f898a031fd8fd143e32a266a8d8f87b: Status 404 returned error can't find the container with id efe9605abd9125d93862e36dcd746fd86f898a031fd8fd143e32a266a8d8f87b Nov 02 06:51:59.679546 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.679512 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" event={"ID":"b2dd1a73-8743-4298-b812-eb823cede79d","Type":"ContainerStarted","Data":"2003cb2c867a25c6b969d8b208ca1832dcb08d84f801436b4400f6fcb1497006"} Nov 02 06:51:59.680007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.679579 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:51:59.680007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.679604 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" event={"ID":"b2dd1a73-8743-4298-b812-eb823cede79d","Type":"ContainerStarted","Data":"efe9605abd9125d93862e36dcd746fd86f898a031fd8fd143e32a266a8d8f87b"} Nov 02 06:51:59.681239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.681212 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" event={"ID":"6d438b90-6711-4ce0-bb13-5c3684e8b4b2","Type":"ContainerStarted","Data":"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c"} Nov 02 06:51:59.681334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.681256 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" event={"ID":"6d438b90-6711-4ce0-bb13-5c3684e8b4b2","Type":"ContainerStarted","Data":"e74719556b95dd0b4ffe4d46b29159c23f4f6cc34942e858a32ea26168cce639"} Nov 02 06:51:59.681334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.681288 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:51:59.789155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.789103 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" podStartSLOduration=1.789091366 podStartE2EDuration="1.789091366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:51:59.787271866 +0000 UTC m=+2073.760606550" watchObservedRunningTime="2026-11-02 06:51:59.789091366 +0000 UTC m=+2073.762426049" Nov 02 06:51:59.897563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:51:59.897518 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" podStartSLOduration=1.897504885 podStartE2EDuration="1.897504885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:51:59.895379546 +0000 UTC m=+2073.868714231" watchObservedRunningTime="2026-11-02 06:51:59.897504885 +0000 UTC m=+2073.870839569" Nov 02 06:52:01.952586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:01.952543 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:52:01.953130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:01.952815 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" podUID="b59768cf-940c-4121-b475-7e7b718d8e3a" containerName="maas-api" containerID="cri-o://22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e" gracePeriod=30 Nov 02 06:52:02.238648 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.238622 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.372758 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\" (UniqueName: \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\") pod \"b59768cf-940c-4121-b475-7e7b718d8e3a\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.372893 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\") pod \"b59768cf-940c-4121-b475-7e7b718d8e3a\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.373046 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") pod \"b59768cf-940c-4121-b475-7e7b718d8e3a\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.373192 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\") pod \"b59768cf-940c-4121-b475-7e7b718d8e3a\" (UID: \"b59768cf-940c-4121-b475-7e7b718d8e3a\") " Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.373654 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca" pod "b59768cf-940c-4121-b475-7e7b718d8e3a" (UID: "b59768cf-940c-4121-b475-7e7b718d8e3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:02.376007 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.373894 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59768cf-940c-4121-b475-7e7b718d8e3a-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:02.376465 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.376445 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs" pod "b59768cf-940c-4121-b475-7e7b718d8e3a" (UID: "b59768cf-940c-4121-b475-7e7b718d8e3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:02.377815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.377781 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x" pod "b59768cf-940c-4121-b475-7e7b718d8e3a" (UID: "b59768cf-940c-4121-b475-7e7b718d8e3a"). InnerVolumeSpecName "kube-api-access-j4p7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:02.377955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.377852 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls" pod "b59768cf-940c-4121-b475-7e7b718d8e3a" (UID: "b59768cf-940c-4121-b475-7e7b718d8e3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:02.475219 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.475144 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4p7x\" (UniqueName: \"kubernetes.io/projected/b59768cf-940c-4121-b475-7e7b718d8e3a-kube-api-access-j4p7x\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:02.475219 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.475172 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:02.475219 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.475181 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59768cf-940c-4121-b475-7e7b718d8e3a-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:02.702594 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.702551 6273 generic.go:356] "Generic (PLEG): container finished" podID="b59768cf-940c-4121-b475-7e7b718d8e3a" containerID="22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e" exitCode=0 Nov 02 06:52:02.702753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.702603 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" Nov 02 06:52:02.702753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.702631 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" event={"ID":"b59768cf-940c-4121-b475-7e7b718d8e3a","Type":"ContainerDied","Data":"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e"} Nov 02 06:52:02.702753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.702685 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m" event={"ID":"b59768cf-940c-4121-b475-7e7b718d8e3a","Type":"ContainerDied","Data":"eda05d86264b356c919bf992867e3b9d75c8586640557eb5ba9db834ce383f74"} Nov 02 06:52:02.702753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.702708 6273 scope.go:122] "RemoveContainer" containerID="22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e" Nov 02 06:52:02.713842 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.713821 6273 scope.go:122] "RemoveContainer" containerID="22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e" Nov 02 06:52:02.714208 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:02.714184 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e\": container with ID starting with 22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e not found: ID does not exist" containerID="22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e" Nov 02 06:52:02.714278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.714216 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e"} err="failed to get container status \"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e\": rpc error: code = NotFound desc = could not find container \"22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e\": container with ID starting with 22618325367cf5ae53f21beb1024d41f114c66c2d9a61c39d0556dbedb648f7e not found: ID does not exist" Nov 02 06:52:02.861052 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.860952 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:52:02.864716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:02.864686 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-gmd8m"] Nov 02 06:52:04.180385 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:04.180308 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59768cf-940c-4121-b475-7e7b718d8e3a" path="/var/lib/kubelet/pods/b59768cf-940c-4121-b475-7e7b718d8e3a/volumes" Nov 02 06:52:05.816812 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.816781 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:52:05.817589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.817568 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b59768cf-940c-4121-b475-7e7b718d8e3a" containerName="maas-api" Nov 02 06:52:05.817589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.817589 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59768cf-940c-4121-b475-7e7b718d8e3a" containerName="maas-api" Nov 02 06:52:05.817805 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.817759 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59768cf-940c-4121-b475-7e7b718d8e3a" containerName="maas-api" Nov 02 06:52:05.823379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.823354 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.825958 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826071 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826106 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826164 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h22f\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826219 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826264 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826289 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826338 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.826586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.826389 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.893737 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.893704 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f42795-openshift-default-dockercfg-9vcqz\"" Nov 02 06:52:05.927891 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.927848 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.927974 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928009 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928043 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h22f\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928124 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928089 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928154 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928179 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928219 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928263 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928370 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928328 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928559 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928751 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928728 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.928851 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.928827 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.929165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.929140 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.930956 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.930933 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.931335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.931313 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.934706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.934686 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:52:05.981829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.981803 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:05.986414 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:05.986387 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h22f\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\") pod \"e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:06.161208 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.161183 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:06.593170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.593089 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:52:06.598863 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:52:06.598835 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae3a01a_11a0_47e1_b147_5ced912bee0c.slice/crio-aa00a1f73750421533756a2ea29f4fd2c09c18629395962ebea9ec9a07ee6f83 WatchSource:0}: Error finding container aa00a1f73750421533756a2ea29f4fd2c09c18629395962ebea9ec9a07ee6f83: Status 404 returned error can't find the container with id aa00a1f73750421533756a2ea29f4fd2c09c18629395962ebea9ec9a07ee6f83 Nov 02 06:52:06.601319 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.601287 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:06.601406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.601353 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:06.601406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.601383 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:06.740800 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.740753 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerStarted","Data":"a1fabb84cfc45da8e411c488f51cb4cc4a7b10b615f81ab21ddbcb5e53f114b3"} Nov 02 06:52:06.741027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.740824 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerStarted","Data":"aa00a1f73750421533756a2ea29f4fd2c09c18629395962ebea9ec9a07ee6f83"} Nov 02 06:52:06.855379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:06.855287 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podStartSLOduration=1.8552730240000002 podStartE2EDuration="1.855273024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:06.854750425 +0000 UTC m=+2080.828085110" watchObservedRunningTime="2026-11-02 06:52:06.855273024 +0000 UTC m=+2080.828607707" Nov 02 06:52:07.161690 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:07.161655 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:07.168620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:07.168591 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:07.747763 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:07.747729 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:07.748696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:07.748676 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:52:08.554439 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:08.554405 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:52:09.004571 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:09.004540 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:52:10.691863 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:10.691807 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 06:52:10.692373 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:10.691884 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 06:52:13.881845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.881810 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:52:13.888339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.888313 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:13.939857 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.939825 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:52:13.970840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.962926 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcsw\" (UniqueName: \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:13.987010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.975301 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:13.987401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.986308 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f42795-dockercfg-9fqcm\"" Nov 02 06:52:13.987549 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:13.986383 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f42795\"" Nov 02 06:52:14.065220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.065182 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:52:14.073443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.073406 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.088265 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.088233 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcsw\" (UniqueName: \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.088435 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.088406 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.089293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.089266 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.113426 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.113375 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:52:14.142547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.142488 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcsw\" (UniqueName: \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\") pod \"payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.189142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.189114 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.189280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.189146 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzdr\" (UniqueName: \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.223499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.223474 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.290830 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.290779 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.291074 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.290837 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzdr\" (UniqueName: \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.291721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.291695 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.349063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.348999 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzdr\" (UniqueName: \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\") pod \"payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.390367 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.390342 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:14.626363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.626330 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:52:14.632291 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:52:14.632266 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967a17e3_bc9c_4528_b4c6_8ba1883905df.slice/crio-12b10b16a285b7762472d0056dcd64d96bb7ac896833b34d7e4a933e6f9e206b WatchSource:0}: Error finding container 12b10b16a285b7762472d0056dcd64d96bb7ac896833b34d7e4a933e6f9e206b: Status 404 returned error can't find the container with id 12b10b16a285b7762472d0056dcd64d96bb7ac896833b34d7e4a933e6f9e206b Nov 02 06:52:14.770024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.769974 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:52:14.775740 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:52:14.775713 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6926d5_1b5b_452d_ad88_d6a6a5b26a3f.slice/crio-26a07884071dbe9f5d1f2c93cded5960ddd52ceefab6c8f4e50e4da25b313783 WatchSource:0}: Error finding container 26a07884071dbe9f5d1f2c93cded5960ddd52ceefab6c8f4e50e4da25b313783: Status 404 returned error can't find the container with id 26a07884071dbe9f5d1f2c93cded5960ddd52ceefab6c8f4e50e4da25b313783 Nov 02 06:52:14.797234 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.797208 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" event={"ID":"967a17e3-bc9c-4528-b4c6-8ba1883905df","Type":"ContainerStarted","Data":"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870"} Nov 02 06:52:14.797353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.797266 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" event={"ID":"967a17e3-bc9c-4528-b4c6-8ba1883905df","Type":"ContainerStarted","Data":"12b10b16a285b7762472d0056dcd64d96bb7ac896833b34d7e4a933e6f9e206b"} Nov 02 06:52:14.797353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.797300 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:14.798869 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:14.798843 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" event={"ID":"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f","Type":"ContainerStarted","Data":"26a07884071dbe9f5d1f2c93cded5960ddd52ceefab6c8f4e50e4da25b313783"} Nov 02 06:52:15.806629 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:15.806585 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" event={"ID":"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f","Type":"ContainerStarted","Data":"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4"} Nov 02 06:52:15.807249 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:15.806821 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:15.914421 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:15.914362 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" podStartSLOduration=2.914344526 podStartE2EDuration="2.914344526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:14.916276534 +0000 UTC m=+2088.889611218" watchObservedRunningTime="2026-11-02 06:52:15.914344526 +0000 UTC m=+2089.887679217" Nov 02 06:52:15.914787 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:15.914756 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" podStartSLOduration=2.914748757 podStartE2EDuration="2.914748757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:15.913778686 +0000 UTC m=+2089.887113371" watchObservedRunningTime="2026-11-02 06:52:15.914748757 +0000 UTC m=+2089.888083441" Nov 02 06:52:16.695196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:16.695166 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:52:16.795067 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:16.795032 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:52:21.695629 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:21.695592 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" containerID="cri-o://f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75" gracePeriod=25 Nov 02 06:52:21.795386 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:21.795347 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" containerID="cri-o://ad92f5e10e0059d57355b54ffb0e33f893c41f454428348068ce88548135454d" gracePeriod=25 Nov 02 06:52:22.542066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:22.542033 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:52:22.666430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:22.666399 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:52:24.026239 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.026189 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" containerID="cri-o://0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c" gracePeriod=45 Nov 02 06:52:24.323164 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.323138 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:52:24.450388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450357 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\" (UniqueName: \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.450607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450421 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.450607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450506 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.450607 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450554 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.450820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450666 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.450820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.450744 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\") pod \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\" (UID: \"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62\") " Nov 02 06:52:24.451357 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.451308 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.451551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.451527 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.451910 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.451882 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.453864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.453834 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.454262 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.454234 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "kube-api-access-m9dx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:24.455407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.455382 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs" pod "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" (UID: "1cfa8c44-a6b4-40ee-aaf1-8950f4762b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:24.552937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552868 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.552937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552895 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dx8\" (UniqueName: \"kubernetes.io/projected/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-kube-api-access-m9dx8\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.552937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552908 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.552937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552920 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.552937 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552932 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.553211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.552945 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.885160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.885074 6273 generic.go:356] "Generic (PLEG): container finished" podID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerID="0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c" exitCode=0 Nov 02 06:52:24.885160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.885126 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" Nov 02 06:52:24.885401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.885158 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" event={"ID":"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62","Type":"ContainerDied","Data":"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c"} Nov 02 06:52:24.885401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.885212 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg" event={"ID":"1cfa8c44-a6b4-40ee-aaf1-8950f4762b62","Type":"ContainerDied","Data":"1e48f02af00e31b4c9f90e2d7d1431ced7b50f36e433ef244ef1cdf704363b89"} Nov 02 06:52:24.885401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.885235 6273 scope.go:122] "RemoveContainer" containerID="0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c" Nov 02 06:52:24.902715 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.902683 6273 scope.go:122] "RemoveContainer" containerID="0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c" Nov 02 06:52:24.903115 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:24.903083 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c\": container with ID starting with 0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c not found: ID does not exist" containerID="0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c" Nov 02 06:52:24.903222 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:24.903124 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c"} err="failed to get container status \"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c\": rpc error: code = NotFound desc = could not find container \"0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c\": container with ID starting with 0c6af89fdb8f0f07c0a87de5c1cc2508731bd635246c8226bf3580bfcef16a0c not found: ID does not exist" Nov 02 06:52:25.041867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:25.041829 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:52:25.047674 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:25.047649 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/premium-w0-bdcd80-kserve-79866d64bf-pfzlg"] Nov 02 06:52:25.810640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:25.810610 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:52:26.181372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:26.181343 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" path="/var/lib/kubelet/pods/1cfa8c44-a6b4-40ee-aaf1-8950f4762b62/volumes" Nov 02 06:52:26.816093 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:26.816070 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:52:27.542720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.542676 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" podUID="13ee9498-d9b9-47bd-8246-ec5d82c79605" containerName="payload-processing" containerID="cri-o://d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8" gracePeriod=25 Nov 02 06:52:27.667252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.667199 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" podUID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" containerName="payload-pre-processing" containerID="cri-o://febb4bea916890eca44ec573fcd2cee3f6f8d571c515d5e1dda63e6729288c88" gracePeriod=25 Nov 02 06:52:27.815704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.815676 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:52:27.899916 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.899881 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\") pod \"13ee9498-d9b9-47bd-8246-ec5d82c79605\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " Nov 02 06:52:27.900108 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.900007 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\" (UniqueName: \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\") pod \"13ee9498-d9b9-47bd-8246-ec5d82c79605\" (UID: \"13ee9498-d9b9-47bd-8246-ec5d82c79605\") " Nov 02 06:52:27.901561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.901528 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume" pod "13ee9498-d9b9-47bd-8246-ec5d82c79605" (UID: "13ee9498-d9b9-47bd-8246-ec5d82c79605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:27.904496 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.904467 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb" pod "13ee9498-d9b9-47bd-8246-ec5d82c79605" (UID: "13ee9498-d9b9-47bd-8246-ec5d82c79605"). InnerVolumeSpecName "kube-api-access-pm6sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:27.908356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.908333 6273 generic.go:356] "Generic (PLEG): container finished" podID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" containerID="febb4bea916890eca44ec573fcd2cee3f6f8d571c515d5e1dda63e6729288c88" exitCode=0 Nov 02 06:52:27.908528 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.908499 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" event={"ID":"d35ea7e6-30fa-43b2-87b5-d4bf469ed961","Type":"ContainerDied","Data":"febb4bea916890eca44ec573fcd2cee3f6f8d571c515d5e1dda63e6729288c88"} Nov 02 06:52:27.910160 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.910135 6273 generic.go:356] "Generic (PLEG): container finished" podID="13ee9498-d9b9-47bd-8246-ec5d82c79605" containerID="d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8" exitCode=0 Nov 02 06:52:27.910270 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.910255 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" event={"ID":"13ee9498-d9b9-47bd-8246-ec5d82c79605","Type":"ContainerDied","Data":"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8"} Nov 02 06:52:27.910315 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.910301 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" event={"ID":"13ee9498-d9b9-47bd-8246-ec5d82c79605","Type":"ContainerDied","Data":"6989f31da6e379229d87dac920e55b1024e093fcb7d06ce5aafaa50e7e245e00"} Nov 02 06:52:27.910353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.910335 6273 scope.go:122] "RemoveContainer" containerID="d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8" Nov 02 06:52:27.910483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.910471 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl" Nov 02 06:52:27.933485 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.933464 6273 scope.go:122] "RemoveContainer" containerID="d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8" Nov 02 06:52:27.933727 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:27.933706 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8\": container with ID starting with d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8 not found: ID does not exist" containerID="d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8" Nov 02 06:52:27.933829 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.933741 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8"} err="failed to get container status \"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8\": rpc error: code = NotFound desc = could not find container \"d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8\": container with ID starting with d9035bf15ca4f9dbfa88fc04eedf1bcc9105a9154eef11baafce4be822c8f5a8 not found: ID does not exist" Nov 02 06:52:27.944557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:27.944534 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:52:28.001616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.001592 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee9498-d9b9-47bd-8246-ec5d82c79605-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:28.001616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.001616 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6sb\" (UniqueName: \"kubernetes.io/projected/13ee9498-d9b9-47bd-8246-ec5d82c79605-kube-api-access-pm6sb\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:28.034390 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.034356 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:52:28.041001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.040961 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e233a7-6d9b76bc9f-khlzl"] Nov 02 06:52:28.102256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.102186 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\") pod \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " Nov 02 06:52:28.102380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.102261 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\" (UniqueName: \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\") pod \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\" (UID: \"d35ea7e6-30fa-43b2-87b5-d4bf469ed961\") " Nov 02 06:52:28.102559 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.102535 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume" pod "d35ea7e6-30fa-43b2-87b5-d4bf469ed961" (UID: "d35ea7e6-30fa-43b2-87b5-d4bf469ed961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:28.102915 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.102898 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:28.104469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.104449 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v" pod "d35ea7e6-30fa-43b2-87b5-d4bf469ed961" (UID: "d35ea7e6-30fa-43b2-87b5-d4bf469ed961"). InnerVolumeSpecName "kube-api-access-sqz6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:28.188203 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.188166 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ee9498-d9b9-47bd-8246-ec5d82c79605" path="/var/lib/kubelet/pods/13ee9498-d9b9-47bd-8246-ec5d82c79605/volumes" Nov 02 06:52:28.204116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.204080 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqz6v\" (UniqueName: \"kubernetes.io/projected/d35ea7e6-30fa-43b2-87b5-d4bf469ed961-kube-api-access-sqz6v\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:28.919083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.919054 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" event={"ID":"d35ea7e6-30fa-43b2-87b5-d4bf469ed961","Type":"ContainerDied","Data":"9ed4ce70be56654d499dc268d9ab1bcb09fa4bc15607833b1eb10eb9a25c88d0"} Nov 02 06:52:28.919083 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.919083 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq" Nov 02 06:52:28.919620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:28.919115 6273 scope.go:122] "RemoveContainer" containerID="febb4bea916890eca44ec573fcd2cee3f6f8d571c515d5e1dda63e6729288c88" Nov 02 06:52:29.076056 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:29.076024 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:52:29.122472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:29.122444 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e233a7-bbb558759-lv5sq"] Nov 02 06:52:30.027237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:30.027193 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:52:30.027237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:30.027229 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:52:30.193923 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:30.186089 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" path="/var/lib/kubelet/pods/d35ea7e6-30fa-43b2-87b5-d4bf469ed961/volumes" Nov 02 06:52:38.553842 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:38.553807 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:52:40.026406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:40.026359 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:52:40.026406 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:40.026382 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:52:41.969309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:41.969238 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:52:42.022275 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.022233 6273 generic.go:356] "Generic (PLEG): container finished" podID="85e99635-ca67-4650-a3fb-87605bf91888" containerID="ad92f5e10e0059d57355b54ffb0e33f893c41f454428348068ce88548135454d" exitCode=0 Nov 02 06:52:42.022601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.022358 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" event={"ID":"85e99635-ca67-4650-a3fb-87605bf91888","Type":"ContainerDied","Data":"ad92f5e10e0059d57355b54ffb0e33f893c41f454428348068ce88548135454d"} Nov 02 06:52:42.024113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.024085 6273 generic.go:356] "Generic (PLEG): container finished" podID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerID="f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75" exitCode=0 Nov 02 06:52:42.024218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.024174 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" event={"ID":"0c82080f-1376-447a-ae5d-dc9bf859db25","Type":"ContainerDied","Data":"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75"} Nov 02 06:52:42.024218 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.024209 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" event={"ID":"0c82080f-1376-447a-ae5d-dc9bf859db25","Type":"ContainerDied","Data":"0d094ee166cb0edcda0a498ab5c1a4f2db350703a37bd2061c5d50549535d2a0"} Nov 02 06:52:42.024331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.024236 6273 scope.go:122] "RemoveContainer" containerID="f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75" Nov 02 06:52:42.024389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.024381 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw" Nov 02 06:52:42.034720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.034696 6273 scope.go:122] "RemoveContainer" containerID="f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75" Nov 02 06:52:42.035043 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:52:42.035017 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75\": container with ID starting with f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75 not found: ID does not exist" containerID="f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75" Nov 02 06:52:42.035125 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.035054 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75"} err="failed to get container status \"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75\": rpc error: code = NotFound desc = could not find container \"f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75\": container with ID starting with f02b784a365ccca37fcd7b80fc4d5fa5b36440e803a6987779df450a7fb40c75 not found: ID does not exist" Nov 02 06:52:42.064505 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.064470 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\" (UniqueName: \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\") pod \"0c82080f-1376-447a-ae5d-dc9bf859db25\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " Nov 02 06:52:42.064589 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.064527 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\") pod \"0c82080f-1376-447a-ae5d-dc9bf859db25\" (UID: \"0c82080f-1376-447a-ae5d-dc9bf859db25\") " Nov 02 06:52:42.064963 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.064939 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume" pod "0c82080f-1376-447a-ae5d-dc9bf859db25" (UID: "0c82080f-1376-447a-ae5d-dc9bf859db25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:42.065242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.065226 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c82080f-1376-447a-ae5d-dc9bf859db25-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:42.066409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.066387 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh" pod "0c82080f-1376-447a-ae5d-dc9bf859db25" (UID: "0c82080f-1376-447a-ae5d-dc9bf859db25"). InnerVolumeSpecName "kube-api-access-h2rvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:42.066652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.066632 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:52:42.166201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.166173 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\") pod \"85e99635-ca67-4650-a3fb-87605bf91888\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " Nov 02 06:52:42.166327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.166224 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\" (UniqueName: \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\") pod \"85e99635-ca67-4650-a3fb-87605bf91888\" (UID: \"85e99635-ca67-4650-a3fb-87605bf91888\") " Nov 02 06:52:42.166515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.166492 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume" pod "85e99635-ca67-4650-a3fb-87605bf91888" (UID: "85e99635-ca67-4650-a3fb-87605bf91888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:42.166703 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.166690 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e99635-ca67-4650-a3fb-87605bf91888-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:42.166739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.166707 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rvh\" (UniqueName: \"kubernetes.io/projected/0c82080f-1376-447a-ae5d-dc9bf859db25-kube-api-access-h2rvh\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:42.168165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.168145 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m" pod "85e99635-ca67-4650-a3fb-87605bf91888" (UID: "85e99635-ca67-4650-a3fb-87605bf91888"). InnerVolumeSpecName "kube-api-access-6gx4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:42.268575 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.268460 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gx4m\" (UniqueName: \"kubernetes.io/projected/85e99635-ca67-4650-a3fb-87605bf91888-kube-api-access-6gx4m\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:42.433182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.433148 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:52:42.446200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:42.446170 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-bdcd80-95c995fcc-p2bvw"] Nov 02 06:52:43.034269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:43.034194 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" event={"ID":"85e99635-ca67-4650-a3fb-87605bf91888","Type":"ContainerDied","Data":"9b2432f2bd1ba7c544d5f991906381528b3480605bad2a85ccb0644e3fcf2ac4"} Nov 02 06:52:43.034269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:43.034218 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp" Nov 02 06:52:43.034269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:43.034250 6273 scope.go:122] "RemoveContainer" containerID="ad92f5e10e0059d57355b54ffb0e33f893c41f454428348068ce88548135454d" Nov 02 06:52:43.189188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:43.189158 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:52:43.194571 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:43.194545 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bdcd80-66cfbbf9b7-c98cp"] Nov 02 06:52:44.180325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:44.180293 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" path="/var/lib/kubelet/pods/0c82080f-1376-447a-ae5d-dc9bf859db25/volumes" Nov 02 06:52:44.180714 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:44.180700 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e99635-ca67-4650-a3fb-87605bf91888" path="/var/lib/kubelet/pods/85e99635-ca67-4650-a3fb-87605bf91888/volumes" Nov 02 06:52:56.057551 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:56.057516 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:52:56.057969 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:56.057723 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" containerID="cri-o://240694bc6ec8b716b36344fc466e25d17638e84648c2b910d4af1171aa42dff3" gracePeriod=30 Nov 02 06:52:56.072507 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:56.072480 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:52:56.072678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:56.072659 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" containerID="cri-o://c66576b3fe2d6f8a47bbda1c35291ed3b25d7b77d00e7eca7266083b9ee7eebf" gracePeriod=30 Nov 02 06:52:58.149550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:58.149518 6273 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:52:58.150113 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:58.149567 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:52:58.312138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:58.312107 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:52:58.358493 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:58.358452 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:52:58.361215 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:52:58.361183 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:53:00.235866 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.235817 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:53:00.625750 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.625670 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 06:53:00.626436 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626412 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" Nov 02 06:53:00.626436 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626433 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626453 6273 cpu_manager.go:414] "containerMap: removing container" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626458 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626477 6273 cpu_manager.go:414] "containerMap: removing container" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626483 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626494 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" containerName="payload-pre-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626499 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" containerName="payload-pre-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626506 6273 cpu_manager.go:414] "containerMap: removing container" podUID="13ee9498-d9b9-47bd-8246-ec5d82c79605" containerName="payload-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626511 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ee9498-d9b9-47bd-8246-ec5d82c79605" containerName="payload-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626603 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfa8c44-a6b4-40ee-aaf1-8950f4762b62" containerName="main" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626613 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c82080f-1376-447a-ae5d-dc9bf859db25" containerName="payload-processing" Nov 02 06:53:00.626618 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626622 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e99635-ca67-4650-a3fb-87605bf91888" containerName="payload-pre-processing" Nov 02 06:53:00.626936 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626630 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35ea7e6-30fa-43b2-87b5-d4bf469ed961" containerName="payload-pre-processing" Nov 02 06:53:00.626936 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.626636 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ee9498-d9b9-47bd-8246-ec5d82c79605" containerName="payload-processing" Nov 02 06:53:00.630223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.630204 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.679175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.679143 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5a85b0-serving-cert\"" Nov 02 06:53:00.716518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.716477 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.716746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.716542 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.716746 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.716718 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wfh\" (UniqueName: \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.716934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.716916 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.730638 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.730607 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 06:53:00.817979 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.817949 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.818182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.818062 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.818182 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.818102 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.818182 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:00.818111 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-serving-cert: secret "maas-api-e2e-inf-b-5a85b0-serving-cert" not found Nov 02 06:53:00.818329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.818180 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wfh\" (UniqueName: \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.818577 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:00.818560 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls podName:e8a735c0-bc8e-4704-84b6-add16df8fecd nodeName:}" failed. No retries permitted until 2026-11-02 06:53:01.318540071 +0000 UTC m=+2135.291874732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls") pod "maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" (UID: "e8a735c0-bc8e-4704-84b6-add16df8fecd") : secret "maas-api-e2e-inf-b-5a85b0-serving-cert" not found Nov 02 06:53:00.818818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.818799 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.820966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.820943 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:00.872678 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:00.872650 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wfh\" (UniqueName: \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:01.202457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.202424 6273 generic.go:356] "Generic (PLEG): container finished" podID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerID="c66576b3fe2d6f8a47bbda1c35291ed3b25d7b77d00e7eca7266083b9ee7eebf" exitCode=0 Nov 02 06:53:01.202593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.202503 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" event={"ID":"7636358f-0d9e-45c5-b571-ce4fd5cf80a1","Type":"ContainerDied","Data":"c66576b3fe2d6f8a47bbda1c35291ed3b25d7b77d00e7eca7266083b9ee7eebf"} Nov 02 06:53:01.204673 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.204651 6273 generic.go:356] "Generic (PLEG): container finished" podID="5db22104-c2e8-4240-aee6-9074f29d3418" containerID="240694bc6ec8b716b36344fc466e25d17638e84648c2b910d4af1171aa42dff3" exitCode=0 Nov 02 06:53:01.204796 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.204724 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerDied","Data":"240694bc6ec8b716b36344fc466e25d17638e84648c2b910d4af1171aa42dff3"} Nov 02 06:53:01.204888 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.204798 6273 scope.go:122] "RemoveContainer" containerID="944eaeaf2876d4d3e13a668b3db7e7487e96236d951981d91bf9b65f9d170802" Nov 02 06:53:01.344669 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.343156 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:01.349814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.349763 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:01.377163 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.377144 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:53:01.442017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.441975 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:53:01.444037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444017 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444115 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444063 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444188 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444176 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444233 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444205 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444243 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.444278 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444257 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444325 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.444350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444328 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444446 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444430 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444494 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444464 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444501 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.444550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444519 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\") pod \"5db22104-c2e8-4240-aee6-9074f29d3418\" (UID: \"5db22104-c2e8-4240-aee6-9074f29d3418\") " Nov 02 06:53:01.444550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444532 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.444886 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.444860 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.445146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.445129 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.445242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.445148 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.445242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.445158 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db22104-c2e8-4240-aee6-9074f29d3418-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.445242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.445167 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.445242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.445176 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.446518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.446496 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.446620 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.446571 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.446865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.446847 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "kube-api-access-h969f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.446912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.446870 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token" pod "5db22104-c2e8-4240-aee6-9074f29d3418" (UID: "5db22104-c2e8-4240-aee6-9074f29d3418"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.544765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.544739 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:01.546059 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546037 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546128 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546080 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546187 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546265 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546318 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546324 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.546407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546388 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546477 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546461 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546499 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.546537 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546514 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546565 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\") pod \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\" (UID: \"7636358f-0d9e-45c5-b571-ce4fd5cf80a1\") " Nov 02 06:53:01.546713 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546690 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.546771 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546754 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.546870 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.546848 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.547141 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547119 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547144 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db22104-c2e8-4240-aee6-9074f29d3418-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547159 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db22104-c2e8-4240-aee6-9074f29d3418-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547175 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547189 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547202 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547216 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547231 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h969f\" (UniqueName: \"kubernetes.io/projected/5db22104-c2e8-4240-aee6-9074f29d3418-kube-api-access-h969f\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.547252 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.547245 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.548464 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.548438 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.548693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.548675 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "kube-api-access-8qpz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.548764 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.548745 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.548934 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.548920 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy" pod "7636358f-0d9e-45c5-b571-ce4fd5cf80a1" (UID: "7636358f-0d9e-45c5-b571-ce4fd5cf80a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.649533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.649495 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpz8\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-kube-api-access-8qpz8\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.649533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.649535 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.649766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.649552 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.649766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.649566 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7636358f-0d9e-45c5-b571-ce4fd5cf80a1-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.928143 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:01.928105 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 06:53:01.936547 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:53:01.936514 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a735c0_bc8e_4704_84b6_add16df8fecd.slice/crio-cb65e53ae6fee008954e0eecba02460f29059d21e3d18d44184bd486ee1d86bc WatchSource:0}: Error finding container cb65e53ae6fee008954e0eecba02460f29059d21e3d18d44184bd486ee1d86bc: Status 404 returned error can't find the container with id cb65e53ae6fee008954e0eecba02460f29059d21e3d18d44184bd486ee1d86bc Nov 02 06:53:02.212628 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.212594 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" event={"ID":"7636358f-0d9e-45c5-b571-ce4fd5cf80a1","Type":"ContainerDied","Data":"272a460755c350fcd757a95013a811e1f222f85abef40d338cae759da1f5dd86"} Nov 02 06:53:02.212752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.212681 6273 scope.go:122] "RemoveContainer" containerID="c66576b3fe2d6f8a47bbda1c35291ed3b25d7b77d00e7eca7266083b9ee7eebf" Nov 02 06:53:02.212752 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.212608 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc" Nov 02 06:53:02.214333 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.214097 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" event={"ID":"e8a735c0-bc8e-4704-84b6-add16df8fecd","Type":"ContainerStarted","Data":"cb65e53ae6fee008954e0eecba02460f29059d21e3d18d44184bd486ee1d86bc"} Nov 02 06:53:02.215748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.215722 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" Nov 02 06:53:02.215860 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.215750 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml" event={"ID":"5db22104-c2e8-4240-aee6-9074f29d3418","Type":"ContainerDied","Data":"21e04247ddd93f783f10de6651a0ade08c5d6b0709bb3813b314221ec56ead6e"} Nov 02 06:53:02.223533 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.223501 6273 scope.go:122] "RemoveContainer" containerID="240694bc6ec8b716b36344fc466e25d17638e84648c2b910d4af1171aa42dff3" Nov 02 06:53:02.370728 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.370695 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:53:02.377180 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.377156 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-whwfc"] Nov 02 06:53:02.485220 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.485151 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:53:02.494170 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:02.494148 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5mnml"] Nov 02 06:53:03.225138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:03.225053 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" event={"ID":"e8a735c0-bc8e-4704-84b6-add16df8fecd","Type":"ContainerStarted","Data":"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5"} Nov 02 06:53:03.225138 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:03.225132 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:03.340414 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:03.340342 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" podStartSLOduration=2.327874718 podStartE2EDuration="3.340324769s" totalImagesPullingTime="1.012450051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:03.339972788 +0000 UTC m=+2137.313307472" watchObservedRunningTime="2026-11-02 06:53:03.340324769 +0000 UTC m=+2137.313659453" Nov 02 06:53:04.179981 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:04.179950 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" path="/var/lib/kubelet/pods/5db22104-c2e8-4240-aee6-9074f29d3418/volumes" Nov 02 06:53:04.180467 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:04.180451 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" path="/var/lib/kubelet/pods/7636358f-0d9e-45c5-b571-ce4fd5cf80a1/volumes" Nov 02 06:53:08.552794 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:08.552752 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:53:09.238225 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:09.238197 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 06:53:11.686687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.686651 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 06:53:11.687321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687303 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.687382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687323 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.687382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687375 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" Nov 02 06:53:11.687382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687380 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" Nov 02 06:53:11.687491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687481 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.687525 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.687502 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7636358f-0d9e-45c5-b571-ce4fd5cf80a1" containerName="istio-proxy" Nov 02 06:53:11.692670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.692650 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.747193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.747167 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 06:53:11.777472 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.777436 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86htx\" (UniqueName: \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.778550 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.778518 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.793799 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.793456 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2e9f81-dockercfg-pz7dm\"" Nov 02 06:53:11.794591 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.794548 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2e9f81\"" Nov 02 06:53:11.844202 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.844164 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 06:53:11.844767 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.844754 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.844840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.844769 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.844938 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.844928 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db22104-c2e8-4240-aee6-9074f29d3418" containerName="istio-proxy" Nov 02 06:53:11.848516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.848496 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:11.879655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.879619 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.879788 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.879738 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86htx\" (UniqueName: \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.880318 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.880299 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.901936 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.901911 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 06:53:11.933871 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.933835 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86htx\" (UniqueName: \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\") pod \"payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:11.980901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.980836 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:11.980901 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:11.980870 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmf4\" (UniqueName: \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.006540 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.006511 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:12.082073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.081949 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.082073 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.082013 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmf4\" (UniqueName: \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.082687 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.082663 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.137717 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.137681 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmf4\" (UniqueName: \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\") pod \"payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.163906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.163881 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:12.383350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.383274 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 06:53:12.389610 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:53:12.389583 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e60dab_0aaa_4b85_8a50_9b031f9b0fcb.slice/crio-581799d676394fcda04e403e66bf6c3e64a3d61a9409628f15086ed6214de9dc WatchSource:0}: Error finding container 581799d676394fcda04e403e66bf6c3e64a3d61a9409628f15086ed6214de9dc: Status 404 returned error can't find the container with id 581799d676394fcda04e403e66bf6c3e64a3d61a9409628f15086ed6214de9dc Nov 02 06:53:12.537276 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:12.537249 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 06:53:12.542962 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:53:12.542935 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4b46a8_1510_497f_a0fa_043f3b6b7c67.slice/crio-10648f3ce7757765ff011a1d063777b16ffdd6584362bd3323488b2210937f07 WatchSource:0}: Error finding container 10648f3ce7757765ff011a1d063777b16ffdd6584362bd3323488b2210937f07: Status 404 returned error can't find the container with id 10648f3ce7757765ff011a1d063777b16ffdd6584362bd3323488b2210937f07 Nov 02 06:53:13.298560 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.298525 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" event={"ID":"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb","Type":"ContainerStarted","Data":"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73"} Nov 02 06:53:13.299009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.298581 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" event={"ID":"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb","Type":"ContainerStarted","Data":"581799d676394fcda04e403e66bf6c3e64a3d61a9409628f15086ed6214de9dc"} Nov 02 06:53:13.299009 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.298606 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:13.300156 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.300129 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" event={"ID":"fa4b46a8-1510-497f-a0fa-043f3b6b7c67","Type":"ContainerStarted","Data":"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99"} Nov 02 06:53:13.300271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.300175 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" event={"ID":"fa4b46a8-1510-497f-a0fa-043f3b6b7c67","Type":"ContainerStarted","Data":"10648f3ce7757765ff011a1d063777b16ffdd6584362bd3323488b2210937f07"} Nov 02 06:53:13.300271 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.300203 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:13.333391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.333362 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" containerID="cri-o://592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a" gracePeriod=45 Nov 02 06:53:13.378977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.378951 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" containerID="cri-o://83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec" gracePeriod=45 Nov 02 06:53:13.381126 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.381096 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" containerID="cri-o://2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179" gracePeriod=45 Nov 02 06:53:13.408019 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.407945 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" podStartSLOduration=2.407932004 podStartE2EDuration="2.407932004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:13.406615339 +0000 UTC m=+2147.379950026" watchObservedRunningTime="2026-11-02 06:53:13.407932004 +0000 UTC m=+2147.381266686" Nov 02 06:53:13.518017 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.516260 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" podStartSLOduration=2.5162427320000003 podStartE2EDuration="2.516242732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:13.515548282 +0000 UTC m=+2147.488882964" watchObservedRunningTime="2026-11-02 06:53:13.516242732 +0000 UTC m=+2147.489577416" Nov 02 06:53:13.703615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.703579 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:53:13.714747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.714724 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:53:13.724070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.724051 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:53:13.800488 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800461 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.800640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800517 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.800640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800546 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\" (UniqueName: \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.800640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800569 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.800640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800637 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.800881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800669 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.800881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800718 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.800881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800778 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\" (UniqueName: \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.800881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800828 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.800881 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800838 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.800974 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801028 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801053 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801108 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801139 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.801199 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801173 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.801500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801227 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.801500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801266 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.801500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801361 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\" (UniqueName: \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\") pod \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\" (UID: \"b0abeaf2-d06a-44bf-87b9-50261b9c95a8\") " Nov 02 06:53:13.801500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801406 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\") pod \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\" (UID: \"6ba3ef96-6d95-4fa3-963c-d055e6243a80\") " Nov 02 06:53:13.801500 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801476 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\" (UniqueName: \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\") pod \"5380a83d-1c98-4d61-b2f8-dd7777741973\" (UID: \"5380a83d-1c98-4d61-b2f8-dd7777741973\") " Nov 02 06:53:13.801743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801505 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.801743 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.801683 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.802187 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802164 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.802293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802191 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.802293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802206 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.802293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802220 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.802293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802234 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.803032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802810 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.803032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.802824 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.803032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803002 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.803032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803026 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.803510 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803489 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.803584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803535 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "kube-api-access-xr766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:13.803639 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803618 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:13.803806 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.803786 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm" pod "6ba3ef96-6d95-4fa3-963c-d055e6243a80" (UID: "6ba3ef96-6d95-4fa3-963c-d055e6243a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.804382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.804247 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:13.805236 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.805215 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "kube-api-access-76lkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:13.805356 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.805334 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "kube-api-access-487kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:13.805658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.805634 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm" pod "b0abeaf2-d06a-44bf-87b9-50261b9c95a8" (UID: "b0abeaf2-d06a-44bf-87b9-50261b9c95a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.806175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.806158 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs" pod "5380a83d-1c98-4d61-b2f8-dd7777741973" (UID: "5380a83d-1c98-4d61-b2f8-dd7777741973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:13.903640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903603 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903629 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903637 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903646 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487kn\" (UniqueName: \"kubernetes.io/projected/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-kube-api-access-487kn\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903657 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903666 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76lkk\" (UniqueName: \"kubernetes.io/projected/5380a83d-1c98-4d61-b2f8-dd7777741973-kube-api-access-76lkk\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903675 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903684 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr766\" (UniqueName: \"kubernetes.io/projected/6ba3ef96-6d95-4fa3-963c-d055e6243a80-kube-api-access-xr766\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903693 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5380a83d-1c98-4d61-b2f8-dd7777741973-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903700 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3ef96-6d95-4fa3-963c-d055e6243a80-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903709 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0abeaf2-d06a-44bf-87b9-50261b9c95a8-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903717 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5380a83d-1c98-4d61-b2f8-dd7777741973-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.903853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:13.903723 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3ef96-6d95-4fa3-963c-d055e6243a80-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:14.308256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.308224 6273 generic.go:356] "Generic (PLEG): container finished" podID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerID="592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a" exitCode=0 Nov 02 06:53:14.308686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.308300 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" event={"ID":"b0abeaf2-d06a-44bf-87b9-50261b9c95a8","Type":"ContainerDied","Data":"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a"} Nov 02 06:53:14.308686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.308349 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" event={"ID":"b0abeaf2-d06a-44bf-87b9-50261b9c95a8","Type":"ContainerDied","Data":"211b8c28ec4284d0c770c7a4bdde858edea9e6b57de2abfb1b3b8361cbba70ba"} Nov 02 06:53:14.308686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.308348 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9" Nov 02 06:53:14.308686 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.308365 6273 scope.go:122] "RemoveContainer" containerID="592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a" Nov 02 06:53:14.309772 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.309751 6273 generic.go:356] "Generic (PLEG): container finished" podID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerID="2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179" exitCode=0 Nov 02 06:53:14.309879 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.309804 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" Nov 02 06:53:14.309879 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.309815 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" event={"ID":"6ba3ef96-6d95-4fa3-963c-d055e6243a80","Type":"ContainerDied","Data":"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179"} Nov 02 06:53:14.309879 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.309853 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z" event={"ID":"6ba3ef96-6d95-4fa3-963c-d055e6243a80","Type":"ContainerDied","Data":"c2b6d5cf05cc592a47462014e9d3ebc0bd8baef8d3dc0696a59514f3599ef3dc"} Nov 02 06:53:14.311335 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.311313 6273 generic.go:356] "Generic (PLEG): container finished" podID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerID="83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec" exitCode=0 Nov 02 06:53:14.311430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.311355 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" Nov 02 06:53:14.311430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.311377 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" event={"ID":"5380a83d-1c98-4d61-b2f8-dd7777741973","Type":"ContainerDied","Data":"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec"} Nov 02 06:53:14.311430 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.311409 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6" event={"ID":"5380a83d-1c98-4d61-b2f8-dd7777741973","Type":"ContainerDied","Data":"d2bb05bb3c86e730662b5c386a5fe108517df1d5cc77858800d5586e2fa1dd70"} Nov 02 06:53:14.320801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.320781 6273 scope.go:122] "RemoveContainer" containerID="592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a" Nov 02 06:53:14.321244 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:14.321224 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a\": container with ID starting with 592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a not found: ID does not exist" containerID="592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a" Nov 02 06:53:14.321368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.321250 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a"} err="failed to get container status \"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a\": rpc error: code = NotFound desc = could not find container \"592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a\": container with ID starting with 592b41db36a1d1af1bf047fa46ccdb13be0e2de1d36e501d2a0a98906236cf7a not found: ID does not exist" Nov 02 06:53:14.321368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.321268 6273 scope.go:122] "RemoveContainer" containerID="2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179" Nov 02 06:53:14.329623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.329599 6273 scope.go:122] "RemoveContainer" containerID="2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179" Nov 02 06:53:14.329949 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:14.329920 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179\": container with ID starting with 2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179 not found: ID does not exist" containerID="2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179" Nov 02 06:53:14.330063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.329955 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179"} err="failed to get container status \"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179\": rpc error: code = NotFound desc = could not find container \"2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179\": container with ID starting with 2d15cf0aa37b6ae282bb43d6b67614edf4d3b9468d035b94339e9fa8bc0cc179 not found: ID does not exist" Nov 02 06:53:14.330063 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.329972 6273 scope.go:122] "RemoveContainer" containerID="83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec" Nov 02 06:53:14.338251 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.338076 6273 scope.go:122] "RemoveContainer" containerID="83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec" Nov 02 06:53:14.338346 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:14.338326 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec\": container with ID starting with 83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec not found: ID does not exist" containerID="83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec" Nov 02 06:53:14.338387 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.338352 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec"} err="failed to get container status \"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec\": rpc error: code = NotFound desc = could not find container \"83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec\": container with ID starting with 83591a61603df2b98fd0011c3843b8a5536678850c51d1deb75b1abe5aea24ec not found: ID does not exist" Nov 02 06:53:14.464217 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.464163 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:53:14.510383 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.510356 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/premium-w5-dcf9c3-kserve-75bc4c7b59-jwcq6"] Nov 02 06:53:14.579652 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.579622 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:53:14.589984 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.589959 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/simulator-w5-dcf9c3-kserve-6457bc5566-jws8z"] Nov 02 06:53:14.703139 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.703105 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:53:14.710267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:14.710241 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-w5-dcf9c3-kserve-c567b87db-574s9"] Nov 02 06:53:16.180556 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.180524 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" path="/var/lib/kubelet/pods/5380a83d-1c98-4d61-b2f8-dd7777741973/volumes" Nov 02 06:53:16.180974 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.180959 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" path="/var/lib/kubelet/pods/6ba3ef96-6d95-4fa3-963c-d055e6243a80/volumes" Nov 02 06:53:16.181364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.181349 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" path="/var/lib/kubelet/pods/b0abeaf2-d06a-44bf-87b9-50261b9c95a8/volumes" Nov 02 06:53:16.297471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.297439 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:53:16.298111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298093 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" Nov 02 06:53:16.298158 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298114 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" Nov 02 06:53:16.298193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298169 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" Nov 02 06:53:16.298193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298177 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" Nov 02 06:53:16.298193 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298187 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" Nov 02 06:53:16.298285 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298194 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" Nov 02 06:53:16.298326 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298319 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5380a83d-1c98-4d61-b2f8-dd7777741973" containerName="main" Nov 02 06:53:16.298364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298339 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba3ef96-6d95-4fa3-963c-d055e6243a80" containerName="main" Nov 02 06:53:16.298364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.298347 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0abeaf2-d06a-44bf-87b9-50261b9c95a8" containerName="main" Nov 02 06:53:16.303303 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.303284 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.353948 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.353921 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f42795-serving-cert\"" Nov 02 06:53:16.402935 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.402905 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:53:16.432221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.432161 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.432221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.432194 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.432434 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.432410 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.432520 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.432502 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjm7\" (UniqueName: \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.533325 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.533297 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.533454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.533348 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjm7\" (UniqueName: \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.533454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.533393 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.533454 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:16.533409 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-serving-cert: secret "maas-api-e2e-api-a-f42795-serving-cert" not found Nov 02 06:53:16.533609 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:16.533477 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls podName:e2be5f02-a59c-42b4-93c1-a25cd85af574 nodeName:}" failed. No retries permitted until 2026-11-02 06:53:17.033457098 +0000 UTC m=+2151.006791766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls") pod "maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" (UID: "e2be5f02-a59c-42b4-93c1-a25cd85af574") : secret "maas-api-e2e-api-a-f42795-serving-cert" not found Nov 02 06:53:16.533609 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.533418 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.534093 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.534068 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.535584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.535566 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:16.587608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:16.587579 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjm7\" (UniqueName: \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:17.040578 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:17.040545 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:17.042977 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:17.042954 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") pod \"maas-api-e2e-api-a-f42795-654c7f9d86-dssbw\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:17.217088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:17.217059 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:17.591599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:17.591569 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:53:17.598376 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:53:17.598350 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2be5f02_a59c_42b4_93c1_a25cd85af574.slice/crio-ce02e2287561b0ce121c89e46b143a2db188daf74afafadb427cb486389fc942 WatchSource:0}: Error finding container ce02e2287561b0ce121c89e46b143a2db188daf74afafadb427cb486389fc942: Status 404 returned error can't find the container with id ce02e2287561b0ce121c89e46b143a2db188daf74afafadb427cb486389fc942 Nov 02 06:53:18.354730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:18.354697 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" event={"ID":"e2be5f02-a59c-42b4-93c1-a25cd85af574","Type":"ContainerStarted","Data":"ce02e2287561b0ce121c89e46b143a2db188daf74afafadb427cb486389fc942"} Nov 02 06:53:19.362076 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:19.362028 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" event={"ID":"e2be5f02-a59c-42b4-93c1-a25cd85af574","Type":"ContainerStarted","Data":"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002"} Nov 02 06:53:19.362569 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:19.362230 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:19.473601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:19.473549 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" podStartSLOduration=2.607258723 podStartE2EDuration="3.473535549s" totalImagesPullingTime="866.276826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:19.471557221 +0000 UTC m=+2153.444891902" watchObservedRunningTime="2026-11-02 06:53:19.473535549 +0000 UTC m=+2153.446870232" Nov 02 06:53:24.314185 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:24.314156 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 06:53:24.314574 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:24.314232 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 06:53:25.373284 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:25.373256 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:53:35.823976 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:35.823900 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:53:35.868473 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:35.868433 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:53:36.946664 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:36.946631 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:53:38.570711 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:38.570640 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:53:39.268012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.262670 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:53:39.288028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.283260 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.358368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.358335 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:53:39.371204 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.371176 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dcf9c3-openshift-default-dockercfg-7kbz6\"" Nov 02 06:53:39.549089 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.530685 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.574150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.574107 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.574804 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.574779 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.574960 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.574942 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.644499 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.644465 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.674048 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.644806 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdhj\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.674377 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.674340 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.674640 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.674618 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.674774 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.674756 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.778721 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.778779 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.778863 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.778892 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.778923 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.779022 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdhj\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.779075 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.779173 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.779200 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.789015 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.788722 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.818111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.789780 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.818111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.791437 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.818111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.795189 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.818111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.795699 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.864012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.863223 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.878807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.875645 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdhj\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.878807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.876132 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.892606 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.892584 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:39.968020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:39.967190 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:40.640103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.639207 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:53:40.640572 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.640161 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 06:53:40.697028 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.689116 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:53:40.802300 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.802261 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:53:40.814456 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.814422 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:40.814588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.814496 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:40.814588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.814535 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:40.837819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:40.837785 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" event={"ID":"05a7fc46-ed0b-4767-97fb-8b021359bb9d","Type":"ContainerStarted","Data":"53035a80c313a72416c26da67a83393aed2e125476bf01a1d1b0f8a778966bdd"} Nov 02 06:53:41.845877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:41.845836 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" event={"ID":"05a7fc46-ed0b-4767-97fb-8b021359bb9d","Type":"ContainerStarted","Data":"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e"} Nov 02 06:53:41.968391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:41.968353 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:41.973012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:41.970977 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:41.973012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:41.971056 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:42.007429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:42.007375 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podStartSLOduration=3.007359049 podStartE2EDuration="3.007359049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:42.006086639 +0000 UTC m=+2175.979421323" watchObservedRunningTime="2026-11-02 06:53:42.007359049 +0000 UTC m=+2175.980693732" Nov 02 06:53:42.970404 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:42.970363 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:42.971031 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:42.970973 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:43.502807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:43.502733 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:53:43.609548 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:43.607848 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:53:43.968454 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:43.968413 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:43.968744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:43.968714 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:44.974731 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:44.974695 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:44.975432 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:44.975397 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:45.972365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:45.972326 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:45.972668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:45.972638 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:46.968596 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:46.968553 6273 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 02 06:53:46.969110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:46.968622 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 02 06:53:48.350696 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.350665 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:48.351269 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.351106 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:48.352255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.352228 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:53:48.503980 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.503945 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" containerID="cri-o://5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1" gracePeriod=25 Nov 02 06:53:48.608507 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.608414 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" containerID="cri-o://ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234" gracePeriod=25 Nov 02 06:53:48.942780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.942744 6273 generic.go:356] "Generic (PLEG): container finished" podID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerID="a1fabb84cfc45da8e411c488f51cb4cc4a7b10b615f81ab21ddbcb5e53f114b3" exitCode=0 Nov 02 06:53:48.943034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.942875 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerDied","Data":"a1fabb84cfc45da8e411c488f51cb4cc4a7b10b615f81ab21ddbcb5e53f114b3"} Nov 02 06:53:48.943494 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.943473 6273 scope.go:122] "RemoveContainer" containerID="a1fabb84cfc45da8e411c488f51cb4cc4a7b10b615f81ab21ddbcb5e53f114b3" Nov 02 06:53:48.945075 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.945035 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:48.945172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.945113 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:48.945172 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:48.945145 6273 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:49.975801 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:49.975763 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerStarted","Data":"5205a160e688a0a19d73f64c435fcd49b27b2f3213298a88ca68312ca9d7a0ce"} Nov 02 06:53:50.026725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.026685 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:50.026725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.026717 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:50.169651 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.168403 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:53:50.171130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.170791 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:53:50.171130 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.170843 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 06:53:50.845531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.845490 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" containerID="cri-o://215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07" gracePeriod=45 Nov 02 06:53:50.889745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:50.889685 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" containerID="cri-o://f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8" gracePeriod=45 Nov 02 06:53:51.036900 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:51.036856 6273 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b81124_c0e7_40f0_9082_a4875e207882.slice/crio-conmon-f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b81124_c0e7_40f0_9082_a4875e207882.slice/crio-f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8.scope\": RecentStats: unable to find data in memory cache]" Nov 02 06:53:51.179037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.178443 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:53:51.179037 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.178500 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 06:53:51.220350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.220323 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:53:51.224659 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.224637 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293043 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\" (UniqueName: \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293200 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\" (UniqueName: \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293264 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293286 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293313 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293350 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\" (UniqueName: \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293394 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293451 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\" (UniqueName: \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293490 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293559 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293672 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293703 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\") pod \"80b81124-c0e7-40f0-9082-a4875e207882\" (UID: \"80b81124-c0e7-40f0-9082-a4875e207882\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.293805 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\") pod \"c57d7a7c-06ed-4975-aa69-bd2c15037394\" (UID: \"c57d7a7c-06ed-4975-aa69-bd2c15037394\") " Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.294818 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295038 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295229 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295814 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295835 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295850 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.295863 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.296400 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.297025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.296454 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.298955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.298924 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:51.299196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.299159 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.299196 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.299177 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:51.300379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.300325 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "kube-api-access-qvcvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:51.300526 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.300508 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz" pod "c57d7a7c-06ed-4975-aa69-bd2c15037394" (UID: "c57d7a7c-06ed-4975-aa69-bd2c15037394"). InnerVolumeSpecName "kube-api-access-tvqqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:51.303340 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.303319 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm" pod "80b81124-c0e7-40f0-9082-a4875e207882" (UID: "80b81124-c0e7-40f0-9082-a4875e207882"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396802 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b81124-c0e7-40f0-9082-a4875e207882-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396831 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqqz\" (UniqueName: \"kubernetes.io/projected/c57d7a7c-06ed-4975-aa69-bd2c15037394-kube-api-access-tvqqz\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396842 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396851 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b81124-c0e7-40f0-9082-a4875e207882-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396859 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcvw\" (UniqueName: \"kubernetes.io/projected/80b81124-c0e7-40f0-9082-a4875e207882-kube-api-access-qvcvw\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396868 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57d7a7c-06ed-4975-aa69-bd2c15037394-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396877 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.396900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:51.396885 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57d7a7c-06ed-4975-aa69-bd2c15037394-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:52.001825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.001786 6273 generic.go:356] "Generic (PLEG): container finished" podID="80b81124-c0e7-40f0-9082-a4875e207882" containerID="f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8" exitCode=0 Nov 02 06:53:52.001825 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.001833 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" Nov 02 06:53:52.002137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.001874 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" event={"ID":"80b81124-c0e7-40f0-9082-a4875e207882","Type":"ContainerDied","Data":"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8"} Nov 02 06:53:52.002137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.001918 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb" event={"ID":"80b81124-c0e7-40f0-9082-a4875e207882","Type":"ContainerDied","Data":"086a1337b2778d1f02c9e8261c05a823d5e16aa6de3a9293ddd1054801e9bedc"} Nov 02 06:53:52.002137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.001941 6273 scope.go:122] "RemoveContainer" containerID="f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8" Nov 02 06:53:52.004157 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.004127 6273 generic.go:356] "Generic (PLEG): container finished" podID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerID="215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07" exitCode=0 Nov 02 06:53:52.004346 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.004328 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" event={"ID":"c57d7a7c-06ed-4975-aa69-bd2c15037394","Type":"ContainerDied","Data":"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07"} Nov 02 06:53:52.004501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.004362 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" event={"ID":"c57d7a7c-06ed-4975-aa69-bd2c15037394","Type":"ContainerDied","Data":"5c3f7f32d878386a0d4f19059bf1dabb4cfefcba63d40be4660cdea3fb350d3b"} Nov 02 06:53:52.004501 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.004420 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622" Nov 02 06:53:52.013264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.013059 6273 scope.go:122] "RemoveContainer" containerID="f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8" Nov 02 06:53:52.013419 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:52.013365 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8\": container with ID starting with f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8 not found: ID does not exist" containerID="f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8" Nov 02 06:53:52.013419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.013391 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8"} err="failed to get container status \"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8\": rpc error: code = NotFound desc = could not find container \"f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8\": container with ID starting with f8835b72807f17b82c48eb0a4c3c2f41dbc5ebe39fa98dd17ab4b32c9d638eb8 not found: ID does not exist" Nov 02 06:53:52.013419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.013413 6273 scope.go:122] "RemoveContainer" containerID="215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07" Nov 02 06:53:52.021897 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.021880 6273 scope.go:122] "RemoveContainer" containerID="215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07" Nov 02 06:53:52.022184 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:53:52.022164 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07\": container with ID starting with 215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07 not found: ID does not exist" containerID="215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07" Nov 02 06:53:52.022237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.022190 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07"} err="failed to get container status \"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07\": rpc error: code = NotFound desc = could not find container \"215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07\": container with ID starting with 215be7ced829228d3779f63bd322a0433617f431d81027bee7e8d58c5604ec07 not found: ID does not exist" Nov 02 06:53:52.174619 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.174581 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:53:52.175183 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.174655 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 06:53:52.182079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.182055 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:53:52.222623 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.222597 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/simulator-w3-fb1398-kserve-668c65f95c-9qfpb"] Nov 02 06:53:52.329375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.329296 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:53:52.336116 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.336089 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-w3-fb1398-kserve-587c84bd9f-5f622"] Nov 02 06:53:52.747887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:52.747854 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:53:53.163803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:53.163336 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:53:53.163803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:53.163406 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 06:53:54.166025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:54.165332 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:53:54.166025 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:54.165388 6273 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 06:53:54.182515 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:54.182485 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b81124-c0e7-40f0-9082-a4875e207882" path="/var/lib/kubelet/pods/80b81124-c0e7-40f0-9082-a4875e207882/volumes" Nov 02 06:53:54.182949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:54.182935 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" path="/var/lib/kubelet/pods/c57d7a7c-06ed-4975-aa69-bd2c15037394/volumes" Nov 02 06:53:55.166634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:55.166603 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:53:55.167793 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:55.167771 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:53:55.785844 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:55.785813 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:53:55.786095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:53:55.786061 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" containerID="cri-o://9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36" gracePeriod=30 Nov 02 06:54:00.026409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:00.026369 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:54:00.026409 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:00.026398 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:54:01.063354 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.063294 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:54:01.067174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.067150 6273 generic.go:356] "Generic (PLEG): container finished" podID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerID="9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36" exitCode=0 Nov 02 06:54:01.067280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.067190 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" Nov 02 06:54:01.067280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.067225 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerDied","Data":"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36"} Nov 02 06:54:01.067280 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.067274 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt" event={"ID":"23204de3-4d1e-41d6-b50e-cce9218a90af","Type":"ContainerDied","Data":"6e807a11475874d2bb97163c52f9e5c6c829381b0bf89e63bdde5bac855f5c17"} Nov 02 06:54:01.067382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.067296 6273 scope.go:122] "RemoveContainer" containerID="9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36" Nov 02 06:54:01.076577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.076559 6273 scope.go:122] "RemoveContainer" containerID="e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec" Nov 02 06:54:01.085864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.085847 6273 scope.go:122] "RemoveContainer" containerID="9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36" Nov 02 06:54:01.086246 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:01.086228 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36\": container with ID starting with 9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36 not found: ID does not exist" containerID="9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36" Nov 02 06:54:01.086309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.086254 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36"} err="failed to get container status \"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36\": rpc error: code = NotFound desc = could not find container \"9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36\": container with ID starting with 9ac1d6a03b50de3d389f318b1cf33f05548a6c8a4cc9b9efaa33302f188cfc36 not found: ID does not exist" Nov 02 06:54:01.086309 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.086271 6273 scope.go:122] "RemoveContainer" containerID="e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec" Nov 02 06:54:01.086511 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:01.086490 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec\": container with ID starting with e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec not found: ID does not exist" containerID="e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec" Nov 02 06:54:01.086563 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.086521 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec"} err="failed to get container status \"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec\": rpc error: code = NotFound desc = could not find container \"e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec\": container with ID starting with e7471506b5c71721b8b6cec1bbbe35095406adcff4730e69814555f3b46cc3ec not found: ID does not exist" Nov 02 06:54:01.213949 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.213920 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.213964 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214084 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214150 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214138 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214180 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214233 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214287 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.214359 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214297 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214358 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214441 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\") pod \"23204de3-4d1e-41d6-b50e-cce9218a90af\" (UID: \"23204de3-4d1e-41d6-b50e-cce9218a90af\") " Nov 02 06:54:01.214522 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214479 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.214695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214526 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.214695 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214570 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:01.214899 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.214874 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.215062 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.215041 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.215146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.215066 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23204de3-4d1e-41d6-b50e-cce9218a90af-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.215146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.215082 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.215146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.215095 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.215146 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.215109 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.216255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.216233 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.216336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.216319 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.216481 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.216462 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:54:01.216564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.216544 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km" pod "23204de3-4d1e-41d6-b50e-cce9218a90af" (UID: "23204de3-4d1e-41d6-b50e-cce9218a90af"). InnerVolumeSpecName "kube-api-access-kk7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.316241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.316168 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.316241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.316195 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.316241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.316205 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7km\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-kube-api-access-kk7km\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.316241 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.316215 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23204de3-4d1e-41d6-b50e-cce9218a90af-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.482706 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.482672 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:54:01.486818 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:01.486790 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-448tt"] Nov 02 06:54:02.179605 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:02.179574 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" path="/var/lib/kubelet/pods/23204de3-4d1e-41d6-b50e-cce9218a90af/volumes" Nov 02 06:54:03.162051 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162018 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 06:54:03.162658 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162641 6273 cpu_manager.go:414] "containerMap: removing container" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.162700 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162661 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.162700 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162681 6273 cpu_manager.go:414] "containerMap: removing container" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" Nov 02 06:54:03.162700 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162686 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" Nov 02 06:54:03.162803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162705 6273 cpu_manager.go:414] "containerMap: removing container" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.162803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162710 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.162803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162733 6273 cpu_manager.go:414] "containerMap: removing container" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" Nov 02 06:54:03.162803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162738 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" Nov 02 06:54:03.162928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162827 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b81124-c0e7-40f0-9082-a4875e207882" containerName="main" Nov 02 06:54:03.162928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162839 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.162928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162848 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57d7a7c-06ed-4975-aa69-bd2c15037394" containerName="main" Nov 02 06:54:03.162928 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.162856 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="23204de3-4d1e-41d6-b50e-cce9218a90af" containerName="istio-proxy" Nov 02 06:54:03.167624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.167603 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.217802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.217779 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2e9f81-serving-cert\"" Nov 02 06:54:03.266971 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.266937 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 06:54:03.340502 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.340470 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.340701 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.340577 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.340802 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.340763 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.340876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.340856 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zk88\" (UniqueName: \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.442053 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.441955 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.442211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.442058 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.442211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.442105 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zk88\" (UniqueName: \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.442320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.442216 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.442668 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.442646 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.444584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.444561 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.444744 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.444723 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.496718 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.496685 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zk88\" (UniqueName: \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\") pod \"maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:03.782380 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:03.782304 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:04.165123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:04.165090 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 06:54:04.171687 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:54:04.171655 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e652264_bcbd_45f3_9a58_0c42115f5529.slice/crio-6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc WatchSource:0}: Error finding container 6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc: Status 404 returned error can't find the container with id 6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc Nov 02 06:54:05.095721 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:05.095681 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" event={"ID":"8e652264-bcbd-45f3-9a58-0c42115f5529","Type":"ContainerStarted","Data":"6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc"} Nov 02 06:54:06.103324 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:06.103287 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" event={"ID":"8e652264-bcbd-45f3-9a58-0c42115f5529","Type":"ContainerStarted","Data":"2534aa1bc4bbb4ef4d039fdd84889b71429c0b3609e9f05f2eab509cbeed6fe6"} Nov 02 06:54:06.103808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:06.103392 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:06.212583 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:06.212526 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" podStartSLOduration=2.23439029 podStartE2EDuration="3.212513241s" totalImagesPullingTime="978.122951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:06.211217532 +0000 UTC m=+2200.184552216" watchObservedRunningTime="2026-11-02 06:54:06.212513241 +0000 UTC m=+2200.185847918" Nov 02 06:54:08.554217 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:08.554175 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:54:08.780123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.780062 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:54:08.801698 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.801662 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\" (UniqueName: \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\") pod \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " Nov 02 06:54:08.801836 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.801775 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\") pod \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\" (UID: \"2fb43458-8056-4472-89a1-b8a06d3eeb3d\") " Nov 02 06:54:08.802484 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.802454 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume" pod "2fb43458-8056-4472-89a1-b8a06d3eeb3d" (UID: "2fb43458-8056-4472-89a1-b8a06d3eeb3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:08.805090 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.805041 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts" pod "2fb43458-8056-4472-89a1-b8a06d3eeb3d" (UID: "2fb43458-8056-4472-89a1-b8a06d3eeb3d"). InnerVolumeSpecName "kube-api-access-tkxts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:08.850291 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.850268 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:54:08.903068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.903030 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\" (UniqueName: \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\") pod \"4c14fdbc-7eed-4120-8665-4d011bcd965d\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " Nov 02 06:54:08.903219 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.903172 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\") pod \"4c14fdbc-7eed-4120-8665-4d011bcd965d\" (UID: \"4c14fdbc-7eed-4120-8665-4d011bcd965d\") " Nov 02 06:54:08.903694 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.903669 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxts\" (UniqueName: \"kubernetes.io/projected/2fb43458-8056-4472-89a1-b8a06d3eeb3d-kube-api-access-tkxts\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:08.903814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.903697 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb43458-8056-4472-89a1-b8a06d3eeb3d-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:08.903814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.903768 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume" pod "4c14fdbc-7eed-4120-8665-4d011bcd965d" (UID: "4c14fdbc-7eed-4120-8665-4d011bcd965d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:08.905388 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:08.905364 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb" pod "4c14fdbc-7eed-4120-8665-4d011bcd965d" (UID: "4c14fdbc-7eed-4120-8665-4d011bcd965d"). InnerVolumeSpecName "kube-api-access-rqmmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:09.005151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.005128 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c14fdbc-7eed-4120-8665-4d011bcd965d-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:09.005151 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.005151 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmmb\" (UniqueName: \"kubernetes.io/projected/4c14fdbc-7eed-4120-8665-4d011bcd965d-kube-api-access-rqmmb\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:09.124431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.124344 6273 generic.go:356] "Generic (PLEG): container finished" podID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerID="ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234" exitCode=0 Nov 02 06:54:09.124431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.124402 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" Nov 02 06:54:09.124634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.124433 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" event={"ID":"4c14fdbc-7eed-4120-8665-4d011bcd965d","Type":"ContainerDied","Data":"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234"} Nov 02 06:54:09.124634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.124489 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74" event={"ID":"4c14fdbc-7eed-4120-8665-4d011bcd965d","Type":"ContainerDied","Data":"e1569c0811119fa3745839c302295825c43cee61e3ba23de2f541d63a80f7461"} Nov 02 06:54:09.124634 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.124503 6273 scope.go:122] "RemoveContainer" containerID="ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234" Nov 02 06:54:09.125906 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.125879 6273 generic.go:356] "Generic (PLEG): container finished" podID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerID="5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1" exitCode=0 Nov 02 06:54:09.126049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.125932 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" Nov 02 06:54:09.126049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.125935 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" event={"ID":"2fb43458-8056-4472-89a1-b8a06d3eeb3d","Type":"ContainerDied","Data":"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1"} Nov 02 06:54:09.126049 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.125974 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd" event={"ID":"2fb43458-8056-4472-89a1-b8a06d3eeb3d","Type":"ContainerDied","Data":"175a3bf0b9357195ab8ad03879ddc209232d2ca536a837cdea289952a14f8a87"} Nov 02 06:54:09.134587 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.134574 6273 scope.go:122] "RemoveContainer" containerID="ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234" Nov 02 06:54:09.134797 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:09.134775 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234\": container with ID starting with ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234 not found: ID does not exist" containerID="ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234" Nov 02 06:54:09.134867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.134803 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234"} err="failed to get container status \"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234\": rpc error: code = NotFound desc = could not find container \"ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234\": container with ID starting with ef6171f31d71c61210dabe5af00826a0859d632dacc9f208aff90bf191e80234 not found: ID does not exist" Nov 02 06:54:09.134867 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.134819 6273 scope.go:122] "RemoveContainer" containerID="5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1" Nov 02 06:54:09.144249 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.144231 6273 scope.go:122] "RemoveContainer" containerID="5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1" Nov 02 06:54:09.144534 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:09.144516 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1\": container with ID starting with 5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1 not found: ID does not exist" containerID="5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1" Nov 02 06:54:09.144601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.144540 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1"} err="failed to get container status \"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1\": rpc error: code = NotFound desc = could not find container \"5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1\": container with ID starting with 5f0c318d886fbcee74356b015f3f439e2e15e8176c65be7fef5aea9ee8d0fba1 not found: ID does not exist" Nov 02 06:54:09.277708 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.277669 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:54:09.287754 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.287723 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fb1398-7795c7cc9f-5gfzd"] Nov 02 06:54:09.390264 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.390187 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:54:09.394720 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:09.394683 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fb1398-5c8d6794c-tpl74"] Nov 02 06:54:10.180405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:10.180368 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" path="/var/lib/kubelet/pods/2fb43458-8056-4472-89a1-b8a06d3eeb3d/volumes" Nov 02 06:54:10.180814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:10.180799 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" path="/var/lib/kubelet/pods/4c14fdbc-7eed-4120-8665-4d011bcd965d/volumes" Nov 02 06:54:12.117873 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:12.117842 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 06:54:16.049649 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:16.049616 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:54:36.703612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.703577 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:54:36.704266 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704250 6273 cpu_manager.go:414] "containerMap: removing container" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" Nov 02 06:54:36.704327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704268 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" Nov 02 06:54:36.704327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704285 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" Nov 02 06:54:36.704327 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704290 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" Nov 02 06:54:36.704422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704369 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb43458-8056-4472-89a1-b8a06d3eeb3d" containerName="payload-processing" Nov 02 06:54:36.704422 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.704389 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c14fdbc-7eed-4120-8665-4d011bcd965d" containerName="payload-pre-processing" Nov 02 06:54:36.709429 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.709408 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.736821 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.736793 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bldj\" (UniqueName: \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.737022 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.736836 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.790644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.789336 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:54:36.806104 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.806061 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e6fde6-dockercfg-4r7tj\"" Nov 02 06:54:36.834002 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.833952 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e6fde6\"" Nov 02 06:54:36.838732 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.838702 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bldj\" (UniqueName: \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.838884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.838745 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.839431 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.839409 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.863627 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.863595 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:54:36.867942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.867923 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:36.891958 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.891935 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bldj\" (UniqueName: \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\") pod \"payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:36.921693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.921667 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:54:36.940103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.940075 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:36.940248 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:36.940107 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrkl\" (UniqueName: \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.023036 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.022900 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:37.041807 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.041774 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.041931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.041818 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrkl\" (UniqueName: \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.042442 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.042416 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.097057 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.097029 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrkl\" (UniqueName: \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\") pod \"payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.183321 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.183295 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:37.397933 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.397859 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:54:37.403909 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:54:37.403884 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c41e2f8_551c_4914_a518_b525edaa0b16.slice/crio-4cf593e1c997a11e0893b93500f1fcf453bb5a1d6487c3c2925222a6d00e6227 WatchSource:0}: Error finding container 4cf593e1c997a11e0893b93500f1fcf453bb5a1d6487c3c2925222a6d00e6227: Status 404 returned error can't find the container with id 4cf593e1c997a11e0893b93500f1fcf453bb5a1d6487c3c2925222a6d00e6227 Nov 02 06:54:37.559790 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:37.559759 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:54:37.566340 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:54:37.566311 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2265b64_43b4_4e6a_b525_88b389e184f4.slice/crio-bd7cfb5ad737a1d5a1b00f844a97767223956fc366ebb1db66ee76e7357c1eb4 WatchSource:0}: Error finding container bd7cfb5ad737a1d5a1b00f844a97767223956fc366ebb1db66ee76e7357c1eb4: Status 404 returned error can't find the container with id bd7cfb5ad737a1d5a1b00f844a97767223956fc366ebb1db66ee76e7357c1eb4 Nov 02 06:54:38.305878 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.305839 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" event={"ID":"b2265b64-43b4-4e6a-b525-88b389e184f4","Type":"ContainerStarted","Data":"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e"} Nov 02 06:54:38.306365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.305909 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" event={"ID":"b2265b64-43b4-4e6a-b525-88b389e184f4","Type":"ContainerStarted","Data":"bd7cfb5ad737a1d5a1b00f844a97767223956fc366ebb1db66ee76e7357c1eb4"} Nov 02 06:54:38.306365 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.306048 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:38.307471 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.307449 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" event={"ID":"1c41e2f8-551c-4914-a518-b525edaa0b16","Type":"ContainerStarted","Data":"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10"} Nov 02 06:54:38.307582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.307486 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" event={"ID":"1c41e2f8-551c-4914-a518-b525edaa0b16","Type":"ContainerStarted","Data":"4cf593e1c997a11e0893b93500f1fcf453bb5a1d6487c3c2925222a6d00e6227"} Nov 02 06:54:38.307582 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.307513 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:38.403393 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.403356 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:54:38.408137 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.408110 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.432095 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.432049 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" podStartSLOduration=2.43203498 podStartE2EDuration="2.43203498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:38.428723241 +0000 UTC m=+2232.402057937" watchObservedRunningTime="2026-11-02 06:54:38.43203498 +0000 UTC m=+2232.405369664" Nov 02 06:54:38.442578 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.442547 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:54:38.457457 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457431 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a2ba11\"/\"openshift-service-ca.crt\"" Nov 02 06:54:38.457617 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457511 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.457689 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457670 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.457785 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457761 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.457852 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457823 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.457895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457870 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.457939 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.457897 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v44f\" (UniqueName: \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.458475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.458453 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a2ba11\"/\"test-model-a2ba11-kserve-self-signed-certs\"" Nov 02 06:54:38.458475 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.458468 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a2ba11\"/\"default-dockercfg-lzdj6\"" Nov 02 06:54:38.458636 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.458547 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a2ba11\"/\"kube-root-ca.crt\"" Nov 02 06:54:38.554103 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.551651 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" podStartSLOduration=2.551629924 podStartE2EDuration="2.551629924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:38.544198534 +0000 UTC m=+2232.517533222" watchObservedRunningTime="2026-11-02 06:54:38.551629924 +0000 UTC m=+2232.524964610" Nov 02 06:54:38.559243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559132 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559243 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559223 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559253 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559288 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v44f\" (UniqueName: \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559489 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559611 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559728 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.559945 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.559926 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.561237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.560968 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.563593 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.563561 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.563834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.563813 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.565126 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:38.565078 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:54:38.615890 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.615856 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v44f\" (UniqueName: \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\") pod \"test-model-a2ba11-kserve-87dc9c4bc-rwgld\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:38.724058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:38.724028 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:39.096479 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:39.096444 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:54:39.103119 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:54:39.103090 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod140de2de_7a2e_404c_b921_1d989fa29a49.slice/crio-04e8a6776ac38d5c448e4db43da2153fc7639b76f3294d83ac076a6bdbc9675a WatchSource:0}: Error finding container 04e8a6776ac38d5c448e4db43da2153fc7639b76f3294d83ac076a6bdbc9675a: Status 404 returned error can't find the container with id 04e8a6776ac38d5c448e4db43da2153fc7639b76f3294d83ac076a6bdbc9675a Nov 02 06:54:39.315682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:39.315641 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" event={"ID":"140de2de-7a2e-404c-b921-1d989fa29a49","Type":"ContainerStarted","Data":"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5"} Nov 02 06:54:39.315682 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:39.315687 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" event={"ID":"140de2de-7a2e-404c-b921-1d989fa29a49","Type":"ContainerStarted","Data":"04e8a6776ac38d5c448e4db43da2153fc7639b76f3294d83ac076a6bdbc9675a"} Nov 02 06:54:39.447819 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:39.447764 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" podStartSLOduration=1.447744839 podStartE2EDuration="1.447744839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:39.444744842 +0000 UTC m=+2233.418079528" watchObservedRunningTime="2026-11-02 06:54:39.447744839 +0000 UTC m=+2233.421079517" Nov 02 06:54:39.725001 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:39.724897 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:42.264527 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.264491 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 06:54:42.269214 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.269182 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316356 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316484 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316547 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8np42\" (UniqueName: \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316588 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316788 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316757 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.316853 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.316817 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.369441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.369404 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a85b0\"/\"openshift-service-ca.crt\"" Nov 02 06:54:42.369679 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.369663 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a85b0\"/\"kube-root-ca.crt\"" Nov 02 06:54:42.369786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.369745 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a85b0\"/\"default-dockercfg-bs4bj\"" Nov 02 06:54:42.369854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.369787 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a85b0\"/\"test-model-5a85b0-kserve-self-signed-certs\"" Nov 02 06:54:42.372320 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.372294 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 06:54:42.417824 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.417780 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.417843 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.417897 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8np42\" (UniqueName: \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.417940 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418047 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.418040 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418381 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.418076 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.418691 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.418658 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.419061 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.419025 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.419565 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.419543 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.426082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.421072 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.426082 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.422529 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.474174 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.474144 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8np42\" (UniqueName: \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\") pod \"test-model-5a85b0-kserve-6f484988fc-kb29m\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.581854 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.581781 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:42.979067 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:42.979028 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 06:54:42.985684 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:54:42.985655 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eff15b3_d9b1_480c_b43e_795550a9895a.slice/crio-3e58c785bc6e360743e2f6fe87031421cb231bd1ac17ee056e21ed60a25c8871 WatchSource:0}: Error finding container 3e58c785bc6e360743e2f6fe87031421cb231bd1ac17ee056e21ed60a25c8871: Status 404 returned error can't find the container with id 3e58c785bc6e360743e2f6fe87031421cb231bd1ac17ee056e21ed60a25c8871 Nov 02 06:54:43.346030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:43.345915 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" event={"ID":"2eff15b3-d9b1-480c-b43e-795550a9895a","Type":"ContainerStarted","Data":"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707"} Nov 02 06:54:43.346030 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:43.345968 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" event={"ID":"2eff15b3-d9b1-480c-b43e-795550a9895a","Type":"ContainerStarted","Data":"3e58c785bc6e360743e2f6fe87031421cb231bd1ac17ee056e21ed60a25c8871"} Nov 02 06:54:43.459643 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:43.459578 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" podStartSLOduration=1.459559326 podStartE2EDuration="1.459559326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:43.457092211 +0000 UTC m=+2237.430426895" watchObservedRunningTime="2026-11-02 06:54:43.459559326 +0000 UTC m=+2237.432894012" Nov 02 06:54:43.582766 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:43.582731 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:45.608588 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.608550 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:54:45.609110 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.608760 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" podUID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" containerName="maas-api" containerID="cri-o://0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3" gracePeriod=30 Nov 02 06:54:45.883379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.883349 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:54:45.961531 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.961497 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") pod \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " Nov 02 06:54:45.961712 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.961592 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\") pod \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " Nov 02 06:54:45.961755 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.961732 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\") pod \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " Nov 02 06:54:45.961795 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.961766 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\" (UniqueName: \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\") pod \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\" (UID: \"f739d1f2-cf87-4e34-8070-b05b5d4d1563\") " Nov 02 06:54:45.962173 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.962139 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca" pod "f739d1f2-cf87-4e34-8070-b05b5d4d1563" (UID: "f739d1f2-cf87-4e34-8070-b05b5d4d1563"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:45.963884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.963849 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs" pod "f739d1f2-cf87-4e34-8070-b05b5d4d1563" (UID: "f739d1f2-cf87-4e34-8070-b05b5d4d1563"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:45.963884 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.963863 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d" pod "f739d1f2-cf87-4e34-8070-b05b5d4d1563" (UID: "f739d1f2-cf87-4e34-8070-b05b5d4d1563"). InnerVolumeSpecName "kube-api-access-7p66d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:45.964043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:45.963870 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls" pod "f739d1f2-cf87-4e34-8070-b05b5d4d1563" (UID: "f739d1f2-cf87-4e34-8070-b05b5d4d1563"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:46.064020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.063945 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f739d1f2-cf87-4e34-8070-b05b5d4d1563-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:46.064020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.063984 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p66d\" (UniqueName: \"kubernetes.io/projected/f739d1f2-cf87-4e34-8070-b05b5d4d1563-kube-api-access-7p66d\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:46.064020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.064016 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:46.064020 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.064026 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f739d1f2-cf87-4e34-8070-b05b5d4d1563-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:46.366175 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.366140 6273 generic.go:356] "Generic (PLEG): container finished" podID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" containerID="0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3" exitCode=0 Nov 02 06:54:46.366391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.366198 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" event={"ID":"f739d1f2-cf87-4e34-8070-b05b5d4d1563","Type":"ContainerDied","Data":"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3"} Nov 02 06:54:46.366391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.366205 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" Nov 02 06:54:46.366391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.366244 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj" event={"ID":"f739d1f2-cf87-4e34-8070-b05b5d4d1563","Type":"ContainerDied","Data":"c885cb7385930dbde0d1be1a54e2ea1519e99e427e069f0bb3130a152ff11297"} Nov 02 06:54:46.366391 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.366270 6273 scope.go:122] "RemoveContainer" containerID="0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3" Nov 02 06:54:46.379748 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.379722 6273 scope.go:122] "RemoveContainer" containerID="0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3" Nov 02 06:54:46.380320 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:54:46.380294 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3\": container with ID starting with 0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3 not found: ID does not exist" containerID="0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3" Nov 02 06:54:46.380419 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.380327 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3"} err="failed to get container status \"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3\": rpc error: code = NotFound desc = could not find container \"0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3\": container with ID starting with 0bbd7e1e140e4a7cf8b691d1e6096e3ae353fa4c325f65884c071e089db124d3 not found: ID does not exist" Nov 02 06:54:46.523478 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.523368 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:54:46.546379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:46.537460 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dcf9c3-685547b8d6-vbqxj"] Nov 02 06:54:48.179912 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:48.179881 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" path="/var/lib/kubelet/pods/f739d1f2-cf87-4e34-8070-b05b5d4d1563/volumes" Nov 02 06:54:48.724914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:48.724879 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:48.737229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:48.737205 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:49.319263 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:49.319233 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:54:49.319647 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:49.319296 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:54:49.405079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:49.405055 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:54:51.250445 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:51.250411 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:54:52.583272 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:52.583238 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:52.600201 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:52.600170 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:53.425223 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:53.425196 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 06:54:55.151079 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:55.151045 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:54:55.151482 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:54:55.151253 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" containerID="cri-o://e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e" gracePeriod=30 Nov 02 06:55:00.424887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.424822 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:55:00.473085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.473047 6273 generic.go:356] "Generic (PLEG): container finished" podID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerID="e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e" exitCode=0 Nov 02 06:55:00.473279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.473098 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" Nov 02 06:55:00.473279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.473121 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" event={"ID":"05a7fc46-ed0b-4767-97fb-8b021359bb9d","Type":"ContainerDied","Data":"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e"} Nov 02 06:55:00.473279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.473191 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj" event={"ID":"05a7fc46-ed0b-4767-97fb-8b021359bb9d","Type":"ContainerDied","Data":"53035a80c313a72416c26da67a83393aed2e125476bf01a1d1b0f8a778966bdd"} Nov 02 06:55:00.473279 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.473227 6273 scope.go:122] "RemoveContainer" containerID="e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e" Nov 02 06:55:00.483081 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.483059 6273 scope.go:122] "RemoveContainer" containerID="e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e" Nov 02 06:55:00.483349 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:55:00.483331 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e\": container with ID starting with e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e not found: ID does not exist" containerID="e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e" Nov 02 06:55:00.483402 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.483356 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e"} err="failed to get container status \"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e\": rpc error: code = NotFound desc = could not find container \"e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e\": container with ID starting with e93108c802c9d7c35e4612df714135fab2e67d9343c13bdca17c410b0263f34e not found: ID does not exist" Nov 02 06:55:00.533972 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.533940 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534008 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534032 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534168 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534086 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534189 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534260 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534318 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534376 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534359 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534367 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.534680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534382 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.534680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534481 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\") pod \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\" (UID: \"05a7fc46-ed0b-4767-97fb-8b021359bb9d\") " Nov 02 06:55:00.534680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534636 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.534680 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534574 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:00.534932 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.534909 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.535179 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.535158 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.535232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.535187 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.535232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.535204 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.535232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.535219 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.535341 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.535234 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.536395 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.536368 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:00.536599 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.536578 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "kube-api-access-hcdhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.536670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.536597 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.536670 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.536609 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token" pod "05a7fc46-ed0b-4767-97fb-8b021359bb9d" (UID: "05a7fc46-ed0b-4767-97fb-8b021359bb9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.636131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.636099 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.636131 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.636128 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.636329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.636140 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdhj\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-kube-api-access-hcdhj\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.636329 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.636150 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7fc46-ed0b-4767-97fb-8b021359bb9d-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.888043 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.887974 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:55:00.890255 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:00.890228 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-t7wqj"] Nov 02 06:55:02.180902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:02.180866 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" path="/var/lib/kubelet/pods/05a7fc46-ed0b-4767-97fb-8b021359bb9d/volumes" Nov 02 06:55:06.272337 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.272249 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" containerID="cri-o://68b78ec0cfba665594bacdf8a86d06e782d856fb11fcb3798adbb36ef91c9439" gracePeriod=45 Nov 02 06:55:06.380088 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.380052 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": dial tcp 10.134.0.69:8000: connect: connection refused" Nov 02 06:55:06.513363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.513326 6273 generic.go:356] "Generic (PLEG): container finished" podID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerID="68b78ec0cfba665594bacdf8a86d06e782d856fb11fcb3798adbb36ef91c9439" exitCode=0 Nov 02 06:55:06.513363 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.513354 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" event={"ID":"d22a27e0-6ea9-4d49-9d3b-215a10dc074c","Type":"ContainerDied","Data":"68b78ec0cfba665594bacdf8a86d06e782d856fb11fcb3798adbb36ef91c9439"} Nov 02 06:55:06.546845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.546826 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:55:06.612868 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.612839 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613035 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613017 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613097 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613065 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613165 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613145 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\" (UniqueName: \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613242 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613227 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613306 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613260 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.613368 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613304 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\") pod \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\" (UID: \"d22a27e0-6ea9-4d49-9d3b-215a10dc074c\") " Nov 02 06:55:06.613516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613488 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.613644 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613583 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.613822 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613804 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.613898 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613823 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.613898 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.613832 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.615237 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.615212 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:06.615334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.615252 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.615379 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.615342 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5" pod "d22a27e0-6ea9-4d49-9d3b-215a10dc074c" (UID: "d22a27e0-6ea9-4d49-9d3b-215a10dc074c"). InnerVolumeSpecName "kube-api-access-mdfz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:06.715371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.715332 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.715371 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.715374 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.715584 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:06.715389 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfz5\" (UniqueName: \"kubernetes.io/projected/d22a27e0-6ea9-4d49-9d3b-215a10dc074c-kube-api-access-mdfz5\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:07.521414 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:07.521380 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" event={"ID":"d22a27e0-6ea9-4d49-9d3b-215a10dc074c","Type":"ContainerDied","Data":"6d9587377a0c8389956f0b182059b61fae521389776d859dca77a39c4479e515"} Nov 02 06:55:07.521414 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:07.521397 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928" Nov 02 06:55:07.521929 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:07.521430 6273 scope.go:122] "RemoveContainer" containerID="68b78ec0cfba665594bacdf8a86d06e782d856fb11fcb3798adbb36ef91c9439" Nov 02 06:55:07.675155 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:07.675117 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:55:07.721443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:07.721413 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-2-w4-680b7b-kserve-6bc9b9c5d7-5d928"] Nov 02 06:55:08.180967 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:08.180926 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" path="/var/lib/kubelet/pods/d22a27e0-6ea9-4d49-9d3b-215a10dc074c/volumes" Nov 02 06:55:08.553406 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:55:08.553331 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:55:10.257653 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:10.257615 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:55:10.258197 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:10.257884 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" containerID="cri-o://8c3af2b431e6d0a138f802ac4659fb447d929da854bf4fcf2f46834177156014" gracePeriod=30 Nov 02 06:55:12.085441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:12.085404 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:55:13.786012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:13.782239 6273 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:55:13.786012 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:13.782312 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:55:15.611705 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.611582 6273 generic.go:356] "Generic (PLEG): container finished" podID="cd8f252c-df8b-4451-8934-3f753886c065" containerID="8c3af2b431e6d0a138f802ac4659fb447d929da854bf4fcf2f46834177156014" exitCode=0 Nov 02 06:55:15.612401 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.612363 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerDied","Data":"8c3af2b431e6d0a138f802ac4659fb447d929da854bf4fcf2f46834177156014"} Nov 02 06:55:15.612601 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.612584 6273 scope.go:122] "RemoveContainer" containerID="d8ddcc2eb67662a37995dfcf957ee9e219498c8ec0c8fa7da54a600fa3d35832" Nov 02 06:55:15.930905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.930883 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:55:15.954355 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954326 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954365 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954418 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954441 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954491 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954482 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954512 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954534 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954589 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954707 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954646 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\") pod \"cd8f252c-df8b-4451-8934-3f753886c065\" (UID: \"cd8f252c-df8b-4451-8934-3f753886c065\") " Nov 02 06:55:15.954905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954721 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.954905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954745 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.954905 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954764 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.955024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.954910 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.955247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.955166 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.955247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.955196 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.955247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.955211 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.955247 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.955225 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.955482 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.955279 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:15.956876 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.956850 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:15.956964 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.956857 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.957195 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.957173 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "kube-api-access-rwtpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:15.957267 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:15.957218 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo" pod "cd8f252c-df8b-4451-8934-3f753886c065" (UID: "cd8f252c-df8b-4451-8934-3f753886c065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:16.057065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.057029 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.057065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.057057 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtpl\" (UniqueName: \"kubernetes.io/projected/cd8f252c-df8b-4451-8934-3f753886c065-kube-api-access-rwtpl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.057065 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.057067 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8f252c-df8b-4451-8934-3f753886c065-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.057334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.057080 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8f252c-df8b-4451-8934-3f753886c065-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.057334 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.057094 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8f252c-df8b-4451-8934-3f753886c065-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.620902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.620816 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" event={"ID":"cd8f252c-df8b-4451-8934-3f753886c065","Type":"ContainerDied","Data":"63276fa3a4f46d2e0a26f99d0be12f6c1293b50a41e5bf85f10fcb737e499b34"} Nov 02 06:55:16.620902 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.620873 6273 scope.go:122] "RemoveContainer" containerID="8c3af2b431e6d0a138f802ac4659fb447d929da854bf4fcf2f46834177156014" Nov 02 06:55:16.621389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.620832 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr" Nov 02 06:55:16.822050 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.821972 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:55:16.868496 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:16.868457 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-ccgxr"] Nov 02 06:55:18.181616 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:18.181574 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8f252c-df8b-4451-8934-3f753886c065" path="/var/lib/kubelet/pods/cd8f252c-df8b-4451-8934-3f753886c065/volumes" Nov 02 06:55:33.341229 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341190 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:55:33.341786 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341770 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341788 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341799 6273 cpu_manager.go:414] "containerMap: removing container" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341804 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341816 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341821 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341829 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" containerName="maas-api" Nov 02 06:55:33.341833 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341833 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" containerName="maas-api" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341864 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341869 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341970 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22a27e0-6ea9-4d49-9d3b-215a10dc074c" containerName="main" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.341982 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.342005 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a7fc46-ed0b-4767-97fb-8b021359bb9d" containerName="istio-proxy" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.342011 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8f252c-df8b-4451-8934-3f753886c065" containerName="istio-proxy" Nov 02 06:55:33.342072 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.342017 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739d1f2-cf87-4e34-8070-b05b5d4d1563" containerName="maas-api" Nov 02 06:55:33.346447 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.346428 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.442339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.442301 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8bc131\"/\"kube-root-ca.crt\"" Nov 02 06:55:33.442339 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.442301 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8bc131\"/\"openshift-service-ca.crt\"" Nov 02 06:55:33.442557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.442426 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8bc131\"/\"test-model-8bc131-kserve-self-signed-certs\"" Nov 02 06:55:33.442557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.442487 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8bc131\"/\"default-dockercfg-4pwhl\"" Nov 02 06:55:33.444667 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.444647 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:55:33.461317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461288 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.461428 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461402 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwljl\" (UniqueName: \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.461470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461430 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.461470 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461450 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.461577 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461549 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.461612 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.461580 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562529 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562494 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562607 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwljl\" (UniqueName: \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562648 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562678 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562730 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562712 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.562953 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.562764 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.563232 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.563205 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.563389 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.563247 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.563516 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.563483 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.565433 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.565410 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.565856 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.565833 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.616631 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.616567 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwljl\" (UniqueName: \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\") pod \"test-model-8bc131-kserve-569d4948cf-fcdp7\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:33.658494 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:33.658465 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:34.031044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:34.031008 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:55:34.037241 ip-10-0-142-103 kubenswrapper[6273]: W1102 06:55:34.037213 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae24fec7_393e_45e1_b69f_8e8f504f3dec.slice/crio-788b8e31d26f478386c815684eb2b6904a3a48458e0a588914f9054e0e415f47 WatchSource:0}: Error finding container 788b8e31d26f478386c815684eb2b6904a3a48458e0a588914f9054e0e415f47: Status 404 returned error can't find the container with id 788b8e31d26f478386c815684eb2b6904a3a48458e0a588914f9054e0e415f47 Nov 02 06:55:34.738845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:34.738809 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" event={"ID":"ae24fec7-393e-45e1-b69f-8e8f504f3dec","Type":"ContainerStarted","Data":"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48"} Nov 02 06:55:34.738845 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:34.738852 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" event={"ID":"ae24fec7-393e-45e1-b69f-8e8f504f3dec","Type":"ContainerStarted","Data":"788b8e31d26f478386c815684eb2b6904a3a48458e0a588914f9054e0e415f47"} Nov 02 06:55:34.857903 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:34.857852 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" podStartSLOduration=1.857839218 podStartE2EDuration="1.857839218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:55:34.857341249 +0000 UTC m=+2288.830675934" watchObservedRunningTime="2026-11-02 06:55:34.857839218 +0000 UTC m=+2288.831173902" Nov 02 06:55:35.659374 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:35.659343 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:38.553724 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:55:38.553689 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:55:39.593230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:39.593195 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:55:43.659127 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:43.659091 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:43.671918 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:43.671895 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:55:43.804745 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:55:43.804715 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:56:01.775654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:01.775620 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:56:08.552633 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:08.552600 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:56:12.157448 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:12.157415 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:56:12.157942 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:12.157644 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" containerID="cri-o://026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0" gracePeriod=30 Nov 02 06:56:17.441887 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.441817 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:56:17.579109 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579066 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579131 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579159 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579211 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579294 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579287 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579369 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579445 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579479 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.579518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579493 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579534 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\") pod \"67e36a03-cf8c-487d-acea-5e9540769be8\" (UID: \"67e36a03-cf8c-487d-acea-5e9540769be8\") " Nov 02 06:56:17.579716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579681 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.579815 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.579720 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.580044 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.580015 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.580200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.580174 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.580200 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.580201 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.580426 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.580024 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:17.580426 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.580171 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.581547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.581521 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "kube-api-access-klmg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:17.581665 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.581645 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:17.581753 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.581730 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:56:17.581809 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.581768 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy" pod "67e36a03-cf8c-487d-acea-5e9540769be8" (UID: "67e36a03-cf8c-487d-acea-5e9540769be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.681654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681609 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmg5\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-kube-api-access-klmg5\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.681654 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681650 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e36a03-cf8c-487d-acea-5e9540769be8-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.681914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681666 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e36a03-cf8c-487d-acea-5e9540769be8-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.681914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681681 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.681914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681693 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e36a03-cf8c-487d-acea-5e9540769be8-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.681914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:17.681706 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e36a03-cf8c-487d-acea-5e9540769be8-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:18.001734 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.001703 6273 generic.go:356] "Generic (PLEG): container finished" podID="67e36a03-cf8c-487d-acea-5e9540769be8" containerID="026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0" exitCode=0 Nov 02 06:56:18.001900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.001747 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" Nov 02 06:56:18.001900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.001778 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerDied","Data":"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0"} Nov 02 06:56:18.001900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.001828 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8" event={"ID":"67e36a03-cf8c-487d-acea-5e9540769be8","Type":"ContainerDied","Data":"9b419585f3648d7f7e83a3aa733caf21d2566077d5edfd19e7494903cc5489d7"} Nov 02 06:56:18.001900 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.001848 6273 scope.go:122] "RemoveContainer" containerID="026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0" Nov 02 06:56:18.013893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.013879 6273 scope.go:122] "RemoveContainer" containerID="6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f" Nov 02 06:56:18.023877 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.023863 6273 scope.go:122] "RemoveContainer" containerID="026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0" Nov 02 06:56:18.024195 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:18.024165 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0\": container with ID starting with 026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0 not found: ID does not exist" containerID="026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0" Nov 02 06:56:18.024302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.024204 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0"} err="failed to get container status \"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0\": rpc error: code = NotFound desc = could not find container \"026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0\": container with ID starting with 026dc654bf372c623b7b705af4796ee83c6b88d2587c38a7210c40375fda03b0 not found: ID does not exist" Nov 02 06:56:18.024302 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.024228 6273 scope.go:122] "RemoveContainer" containerID="6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f" Nov 02 06:56:18.024430 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:18.024416 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f\": container with ID starting with 6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f not found: ID does not exist" containerID="6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f" Nov 02 06:56:18.024480 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.024438 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f"} err="failed to get container status \"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f\": rpc error: code = NotFound desc = could not find container \"6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f\": container with ID starting with 6ffd5952f0f151eef32333a57504fc7f4b4604f9ce252bcf316d99716a3a011f not found: ID does not exist" Nov 02 06:56:18.157336 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.157308 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:56:18.203405 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:18.203374 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-s28x8"] Nov 02 06:56:20.180832 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:20.180799 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" path="/var/lib/kubelet/pods/67e36a03-cf8c-487d-acea-5e9540769be8/volumes" Nov 02 06:56:30.033557 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:30.033520 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:56:38.553618 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:38.553539 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:56:43.162840 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:43.162799 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:56:43.263931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:43.263896 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:56:48.163624 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.163583 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" podUID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" containerName="payload-processing" containerID="cri-o://8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4" gracePeriod=25 Nov 02 06:56:48.264704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.264660 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" podUID="967a17e3-bc9c-4528-b4c6-8ba1883905df" containerName="payload-pre-processing" containerID="cri-o://a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870" gracePeriod=25 Nov 02 06:56:48.457469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.457436 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:56:48.534123 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.534098 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:56:48.564789 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.564747 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\") pod \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " Nov 02 06:56:48.565045 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.564809 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\" (UniqueName: \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\") pod \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\" (UID: \"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f\") " Nov 02 06:56:48.565230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.565204 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume" pod "5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" (UID: "5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:48.565561 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.565470 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:48.570950 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.568081 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr" pod "5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" (UID: "5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f"). InnerVolumeSpecName "kube-api-access-glzdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:48.666848 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.666812 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\" (UniqueName: \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\") pod \"967a17e3-bc9c-4528-b4c6-8ba1883905df\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " Nov 02 06:56:48.667058 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.666926 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\") pod \"967a17e3-bc9c-4528-b4c6-8ba1883905df\" (UID: \"967a17e3-bc9c-4528-b4c6-8ba1883905df\") " Nov 02 06:56:48.667396 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.667370 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume" pod "967a17e3-bc9c-4528-b4c6-8ba1883905df" (UID: "967a17e3-bc9c-4528-b4c6-8ba1883905df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:48.667439 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.667385 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzdr\" (UniqueName: \"kubernetes.io/projected/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f-kube-api-access-glzdr\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:48.668956 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.668934 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw" pod "967a17e3-bc9c-4528-b4c6-8ba1883905df" (UID: "967a17e3-bc9c-4528-b4c6-8ba1883905df"). InnerVolumeSpecName "kube-api-access-bbcsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:48.768450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.768356 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcsw\" (UniqueName: \"kubernetes.io/projected/967a17e3-bc9c-4528-b4c6-8ba1883905df-kube-api-access-bbcsw\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:48.768450 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:48.768384 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967a17e3-bc9c-4528-b4c6-8ba1883905df-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:49.196085 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.196050 6273 generic.go:356] "Generic (PLEG): container finished" podID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" containerID="8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4" exitCode=0 Nov 02 06:56:49.196492 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.196105 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" Nov 02 06:56:49.196492 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.196136 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" event={"ID":"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f","Type":"ContainerDied","Data":"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4"} Nov 02 06:56:49.196492 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.196185 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx" event={"ID":"5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f","Type":"ContainerDied","Data":"26a07884071dbe9f5d1f2c93cded5960ddd52ceefab6c8f4e50e4da25b313783"} Nov 02 06:56:49.196492 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.196207 6273 scope.go:122] "RemoveContainer" containerID="8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4" Nov 02 06:56:49.197725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.197702 6273 generic.go:356] "Generic (PLEG): container finished" podID="967a17e3-bc9c-4528-b4c6-8ba1883905df" containerID="a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870" exitCode=0 Nov 02 06:56:49.197820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.197753 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" Nov 02 06:56:49.197820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.197768 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" event={"ID":"967a17e3-bc9c-4528-b4c6-8ba1883905df","Type":"ContainerDied","Data":"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870"} Nov 02 06:56:49.197820 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.197799 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp" event={"ID":"967a17e3-bc9c-4528-b4c6-8ba1883905df","Type":"ContainerDied","Data":"12b10b16a285b7762472d0056dcd64d96bb7ac896833b34d7e4a933e6f9e206b"} Nov 02 06:56:49.206895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.206812 6273 scope.go:122] "RemoveContainer" containerID="8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4" Nov 02 06:56:49.207156 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:49.207134 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4\": container with ID starting with 8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4 not found: ID does not exist" containerID="8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4" Nov 02 06:56:49.207230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.207164 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4"} err="failed to get container status \"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4\": rpc error: code = NotFound desc = could not find container \"8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4\": container with ID starting with 8c99103b9ddaa0ab0b7582c6e44f1da7dce85da53a4282fbbfd331816dda17a4 not found: ID does not exist" Nov 02 06:56:49.207230 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.207180 6273 scope.go:122] "RemoveContainer" containerID="a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870" Nov 02 06:56:49.218256 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.218236 6273 scope.go:122] "RemoveContainer" containerID="a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870" Nov 02 06:56:49.218539 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:56:49.218514 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870\": container with ID starting with a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870 not found: ID does not exist" containerID="a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870" Nov 02 06:56:49.218632 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.218544 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870"} err="failed to get container status \"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870\": rpc error: code = NotFound desc = could not find container \"a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870\": container with ID starting with a55102f8f792c2df35a785769328d60dda1e78628337cd7b27fecaac5bef9870 not found: ID does not exist" Nov 02 06:56:49.350311 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.350275 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:56:49.355656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.355630 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f42795-7c75f95b59-56lpp"] Nov 02 06:56:49.459604 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.459508 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:56:49.462353 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:49.462329 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f42795-7fdfbc4ffc-rxlcx"] Nov 02 06:56:50.180010 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:50.179965 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" path="/var/lib/kubelet/pods/5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f/volumes" Nov 02 06:56:50.180441 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:56:50.180419 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967a17e3-bc9c-4528-b4c6-8ba1883905df" path="/var/lib/kubelet/pods/967a17e3-bc9c-4528-b4c6-8ba1883905df/volumes" Nov 02 06:57:05.543350 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:57:05.543308 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:57:08.553745 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:57:08.553708 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:57:38.552526 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:57:38.552481 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:57:40.429659 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:57:40.429625 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:57:59.625403 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:57:59.625366 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:57:59.626032 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:57:59.625609 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" podUID="e2be5f02-a59c-42b4-93c1-a25cd85af574" containerName="maas-api" containerID="cri-o://910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002" gracePeriod=30 Nov 02 06:57:59.899308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:57:59.899284 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:58:00.014690 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.014652 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") pod \"e2be5f02-a59c-42b4-93c1-a25cd85af574\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " Nov 02 06:58:00.014893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.014720 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\") pod \"e2be5f02-a59c-42b4-93c1-a25cd85af574\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " Nov 02 06:58:00.014893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.014831 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\") pod \"e2be5f02-a59c-42b4-93c1-a25cd85af574\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " Nov 02 06:58:00.014893 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.014870 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\" (UniqueName: \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\") pod \"e2be5f02-a59c-42b4-93c1-a25cd85af574\" (UID: \"e2be5f02-a59c-42b4-93c1-a25cd85af574\") " Nov 02 06:58:00.015282 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.015252 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca" pod "e2be5f02-a59c-42b4-93c1-a25cd85af574" (UID: "e2be5f02-a59c-42b4-93c1-a25cd85af574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:58:00.017077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.017052 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7" pod "e2be5f02-a59c-42b4-93c1-a25cd85af574" (UID: "e2be5f02-a59c-42b4-93c1-a25cd85af574"). InnerVolumeSpecName "kube-api-access-mtjm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:00.017178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.017090 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls" pod "e2be5f02-a59c-42b4-93c1-a25cd85af574" (UID: "e2be5f02-a59c-42b4-93c1-a25cd85af574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:58:00.017292 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.017269 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs" pod "e2be5f02-a59c-42b4-93c1-a25cd85af574" (UID: "e2be5f02-a59c-42b4-93c1-a25cd85af574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:58:00.116210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.116172 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be5f02-a59c-42b4-93c1-a25cd85af574-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:00.116210 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.116210 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjm7\" (UniqueName: \"kubernetes.io/projected/e2be5f02-a59c-42b4-93c1-a25cd85af574-kube-api-access-mtjm7\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:00.116407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.116226 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:00.116407 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.116240 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2be5f02-a59c-42b4-93c1-a25cd85af574-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:00.651078 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.651037 6273 generic.go:356] "Generic (PLEG): container finished" podID="e2be5f02-a59c-42b4-93c1-a25cd85af574" containerID="910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002" exitCode=0 Nov 02 06:58:00.651506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.651109 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" Nov 02 06:58:00.651506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.651111 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" event={"ID":"e2be5f02-a59c-42b4-93c1-a25cd85af574","Type":"ContainerDied","Data":"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002"} Nov 02 06:58:00.651506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.651168 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw" event={"ID":"e2be5f02-a59c-42b4-93c1-a25cd85af574","Type":"ContainerDied","Data":"ce02e2287561b0ce121c89e46b143a2db188daf74afafadb427cb486389fc942"} Nov 02 06:58:00.651506 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.651190 6273 scope.go:122] "RemoveContainer" containerID="910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002" Nov 02 06:58:00.661458 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.661435 6273 scope.go:122] "RemoveContainer" containerID="910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002" Nov 02 06:58:00.661791 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:58:00.661768 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002\": container with ID starting with 910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002 not found: ID does not exist" containerID="910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002" Nov 02 06:58:00.661859 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.661800 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002"} err="failed to get container status \"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002\": rpc error: code = NotFound desc = could not find container \"910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002\": container with ID starting with 910f5d453fba414ac459fe7d74ab3649450674b1a983034a7a59fda4a147f002 not found: ID does not exist" Nov 02 06:58:00.808999 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.808950 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:58:00.855375 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:00.855342 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f42795-654c7f9d86-dssbw"] Nov 02 06:58:02.179483 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:02.179449 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2be5f02-a59c-42b4-93c1-a25cd85af574" path="/var/lib/kubelet/pods/e2be5f02-a59c-42b4-93c1-a25cd85af574/volumes" Nov 02 06:58:08.570092 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:58:08.569862 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:58:08.575111 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:08.575081 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:58:08.575308 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:08.575278 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" containerID="cri-o://5205a160e688a0a19d73f64c435fcd49b27b2f3213298a88ca68312ca9d7a0ce" gracePeriod=30 Nov 02 06:58:10.169399 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:10.169355 6273 patch_prober.go:29] interesting pod/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:58:10.169781 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:10.169417 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:58:13.738757 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.738723 6273 generic.go:356] "Generic (PLEG): container finished" podID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerID="5205a160e688a0a19d73f64c435fcd49b27b2f3213298a88ca68312ca9d7a0ce" exitCode=0 Nov 02 06:58:13.739224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.738796 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerDied","Data":"5205a160e688a0a19d73f64c435fcd49b27b2f3213298a88ca68312ca9d7a0ce"} Nov 02 06:58:13.739224 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.738880 6273 scope.go:122] "RemoveContainer" containerID="a1fabb84cfc45da8e411c488f51cb4cc4a7b10b615f81ab21ddbcb5e53f114b3" Nov 02 06:58:13.845895 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.845831 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:58:13.881034 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.880984 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881070 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881117 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881161 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881144 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881167 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881260 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:58:13.881331 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881273 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881363 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881440 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881482 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\") pod \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\" (UID: \"8ae3a01a-11a0-47e1-b147-5ced912bee0c\") " Nov 02 06:58:13.881503 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881487 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:58:13.881725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881499 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:58:13.881725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881490 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:58:13.881725 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881688 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:58:13.881952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881885 6273 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istiod-ca-cert\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.881952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881905 6273 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-data\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.881952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881920 6273 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-credential-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.881952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881934 6273 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.881952 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.881947 6273 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-workload-socket\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.883659 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.883633 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "kube-api-access-8h22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:13.883780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.883650 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:58:13.883780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.883722 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:13.883780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.883738 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy" pod "8ae3a01a-11a0-47e1-b147-5ced912bee0c" (UID: "8ae3a01a-11a0-47e1-b147-5ced912bee0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:58:13.982849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.982809 6273 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-envoy\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.982849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.982846 6273 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-token\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.982849 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.982856 6273 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae3a01a-11a0-47e1-b147-5ced912bee0c-istio-podinfo\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.983121 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:13.982867 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h22f\" (UniqueName: \"kubernetes.io/projected/8ae3a01a-11a0-47e1-b147-5ced912bee0c-kube-api-access-8h22f\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:14.746469 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:14.746438 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" Nov 02 06:58:14.746907 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:14.746445 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv" event={"ID":"8ae3a01a-11a0-47e1-b147-5ced912bee0c","Type":"ContainerDied","Data":"aa00a1f73750421533756a2ea29f4fd2c09c18629395962ebea9ec9a07ee6f83"} Nov 02 06:58:14.746907 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:14.746570 6273 scope.go:122] "RemoveContainer" containerID="5205a160e688a0a19d73f64c435fcd49b27b2f3213298a88ca68312ca9d7a0ce" Nov 02 06:58:14.947518 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:14.947484 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:58:14.967068 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:14.967031 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f42795-openshift-default-56755b98c9-rvmsv"] Nov 02 06:58:16.180211 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:16.180176 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" path="/var/lib/kubelet/pods/8ae3a01a-11a0-47e1-b147-5ced912bee0c/volumes" Nov 02 06:58:20.769747 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:20.769707 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:58:20.823719 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:20.823677 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:58:20.870655 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:20.870615 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:58:25.770221 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:25.770176 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" podUID="b2265b64-43b4-4e6a-b525-88b389e184f4" containerName="payload-processing" containerID="cri-o://8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e" gracePeriod=25 Nov 02 06:58:25.871814 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:25.871757 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" podUID="1c41e2f8-551c-4914-a518-b525edaa0b16" containerName="payload-pre-processing" containerID="cri-o://29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10" gracePeriod=25 Nov 02 06:58:26.037586 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.037523 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:58:26.131530 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.131507 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:58:26.131641 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.131629 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\") pod \"b2265b64-43b4-4e6a-b525-88b389e184f4\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " Nov 02 06:58:26.131693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.131680 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\" (UniqueName: \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\") pod \"b2265b64-43b4-4e6a-b525-88b389e184f4\" (UID: \"b2265b64-43b4-4e6a-b525-88b389e184f4\") " Nov 02 06:58:26.131966 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.131944 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume" pod "b2265b64-43b4-4e6a-b525-88b389e184f4" (UID: "b2265b64-43b4-4e6a-b525-88b389e184f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:58:26.132226 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.132211 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2265b64-43b4-4e6a-b525-88b389e184f4-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:26.133615 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.133594 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl" pod "b2265b64-43b4-4e6a-b525-88b389e184f4" (UID: "b2265b64-43b4-4e6a-b525-88b389e184f4"). InnerVolumeSpecName "kube-api-access-bzrkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:26.233608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.233582 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\" (UniqueName: \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\") pod \"1c41e2f8-551c-4914-a518-b525edaa0b16\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " Nov 02 06:58:26.233765 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.233657 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\") pod \"1c41e2f8-551c-4914-a518-b525edaa0b16\" (UID: \"1c41e2f8-551c-4914-a518-b525edaa0b16\") " Nov 02 06:58:26.234027 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.233983 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume" pod "1c41e2f8-551c-4914-a518-b525edaa0b16" (UID: "1c41e2f8-551c-4914-a518-b525edaa0b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:58:26.234156 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.234100 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrkl\" (UniqueName: \"kubernetes.io/projected/b2265b64-43b4-4e6a-b525-88b389e184f4-kube-api-access-bzrkl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:26.235677 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.235654 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj" pod "1c41e2f8-551c-4914-a518-b525edaa0b16" (UID: "1c41e2f8-551c-4914-a518-b525edaa0b16"). InnerVolumeSpecName "kube-api-access-8bldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:26.335178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.335102 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bldj\" (UniqueName: \"kubernetes.io/projected/1c41e2f8-551c-4914-a518-b525edaa0b16-kube-api-access-8bldj\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:26.335178 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.335126 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c41e2f8-551c-4914-a518-b525edaa0b16-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:26.817455 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.817421 6273 generic.go:356] "Generic (PLEG): container finished" podID="b2265b64-43b4-4e6a-b525-88b389e184f4" containerID="8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e" exitCode=0 Nov 02 06:58:26.817861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.817466 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" Nov 02 06:58:26.817861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.817503 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" event={"ID":"b2265b64-43b4-4e6a-b525-88b389e184f4","Type":"ContainerDied","Data":"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e"} Nov 02 06:58:26.817861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.817555 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl" event={"ID":"b2265b64-43b4-4e6a-b525-88b389e184f4","Type":"ContainerDied","Data":"bd7cfb5ad737a1d5a1b00f844a97767223956fc366ebb1db66ee76e7357c1eb4"} Nov 02 06:58:26.817861 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.817596 6273 scope.go:122] "RemoveContainer" containerID="8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e" Nov 02 06:58:26.818955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.818904 6273 generic.go:356] "Generic (PLEG): container finished" podID="1c41e2f8-551c-4914-a518-b525edaa0b16" containerID="29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10" exitCode=0 Nov 02 06:58:26.818955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.818943 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" Nov 02 06:58:26.818955 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.818951 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" event={"ID":"1c41e2f8-551c-4914-a518-b525edaa0b16","Type":"ContainerDied","Data":"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10"} Nov 02 06:58:26.819142 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.819017 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc" event={"ID":"1c41e2f8-551c-4914-a518-b525edaa0b16","Type":"ContainerDied","Data":"4cf593e1c997a11e0893b93500f1fcf453bb5a1d6487c3c2925222a6d00e6227"} Nov 02 06:58:26.827443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.827426 6273 scope.go:122] "RemoveContainer" containerID="8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e" Nov 02 06:58:26.827689 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:58:26.827668 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e\": container with ID starting with 8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e not found: ID does not exist" containerID="8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e" Nov 02 06:58:26.827739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.827698 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e"} err="failed to get container status \"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e\": rpc error: code = NotFound desc = could not find container \"8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e\": container with ID starting with 8a21677f7749f19ab22504e8963a238a5687964f20b6505795a08f46295a7d2e not found: ID does not exist" Nov 02 06:58:26.827739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.827716 6273 scope.go:122] "RemoveContainer" containerID="29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10" Nov 02 06:58:26.836024 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.836007 6273 scope.go:122] "RemoveContainer" containerID="29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10" Nov 02 06:58:26.836273 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:58:26.836251 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10\": container with ID starting with 29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10 not found: ID does not exist" containerID="29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10" Nov 02 06:58:26.836342 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.836279 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10"} err="failed to get container status \"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10\": rpc error: code = NotFound desc = could not find container \"29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10\": container with ID starting with 29d62aa7669104a34e8037478f086e79973aeee65b938b50824abf46031c1c10 not found: ID does not exist" Nov 02 06:58:26.972466 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:26.972434 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:58:27.018865 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:27.018830 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e6fde6-78fcd54759-hj7hc"] Nov 02 06:58:27.080645 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:27.080575 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:58:27.084704 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:27.084679 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e6fde6-56cd9697f7-v26xl"] Nov 02 06:58:28.179808 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:28.179772 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c41e2f8-551c-4914-a518-b525edaa0b16" path="/var/lib/kubelet/pods/1c41e2f8-551c-4914-a518-b525edaa0b16/volumes" Nov 02 06:58:28.180364 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:28.180344 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2265b64-43b4-4e6a-b525-88b389e184f4" path="/var/lib/kubelet/pods/b2265b64-43b4-4e6a-b525-88b389e184f4/volumes" Nov 02 06:58:38.553020 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:58:38.552970 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:58:45.948716 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:58:45.948682 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:59:08.552670 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:59:08.552629 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:59:15.484066 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:15.484033 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:59:30.504162 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.504103 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" podUID="140de2de-7a2e-404c-b921-1d989fa29a49" containerName="main" containerID="cri-o://f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5" gracePeriod=45 Nov 02 06:59:30.773580 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.773555 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:59:30.903608 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903571 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\" (UniqueName: \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.903799 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903745 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.903872 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903815 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\" (UniqueName: \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.903931 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903893 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.904011 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903939 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.904077 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.903980 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\") pod \"140de2de-7a2e-404c-b921-1d989fa29a49\" (UID: \"140de2de-7a2e-404c-b921-1d989fa29a49\") " Nov 02 06:59:30.904238 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.904206 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:30.904438 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.904417 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:30.904566 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.904542 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:30.904736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.904716 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:30.904736 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.904597 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:30.906106 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.906078 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "kube-api-access-9v44f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:30.906261 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.906242 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:30.906570 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:30.906554 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs" pod "140de2de-7a2e-404c-b921-1d989fa29a49" (UID: "140de2de-7a2e-404c-b921-1d989fa29a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:31.006100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.006029 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:31.006100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.006066 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v44f\" (UniqueName: \"kubernetes.io/projected/140de2de-7a2e-404c-b921-1d989fa29a49-kube-api-access-9v44f\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:31.006100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.006076 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/140de2de-7a2e-404c-b921-1d989fa29a49-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:31.006100 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.006085 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140de2de-7a2e-404c-b921-1d989fa29a49-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:31.178372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.178336 6273 generic.go:356] "Generic (PLEG): container finished" podID="140de2de-7a2e-404c-b921-1d989fa29a49" containerID="f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5" exitCode=0 Nov 02 06:59:31.178372 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.178371 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" Nov 02 06:59:31.178564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.178398 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" event={"ID":"140de2de-7a2e-404c-b921-1d989fa29a49","Type":"ContainerDied","Data":"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5"} Nov 02 06:59:31.178564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.178430 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld" event={"ID":"140de2de-7a2e-404c-b921-1d989fa29a49","Type":"ContainerDied","Data":"04e8a6776ac38d5c448e4db43da2153fc7639b76f3294d83ac076a6bdbc9675a"} Nov 02 06:59:31.178564 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.178453 6273 scope.go:122] "RemoveContainer" containerID="f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5" Nov 02 06:59:31.188120 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.188053 6273 scope.go:122] "RemoveContainer" containerID="f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5" Nov 02 06:59:31.188327 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:59:31.188308 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5\": container with ID starting with f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5 not found: ID does not exist" containerID="f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5" Nov 02 06:59:31.188382 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.188334 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5"} err="failed to get container status \"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5\": rpc error: code = NotFound desc = could not find container \"f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5\": container with ID starting with f53038405f4bfcb710c2469db04395c7c1bb704e234295fd27e190f9ba883ee5 not found: ID does not exist" Nov 02 06:59:31.335739 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.335660 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:59:31.381517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.381480 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a2ba11/test-model-a2ba11-kserve-87dc9c4bc-rwgld"] Nov 02 06:59:31.491443 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:31.491409 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:59:32.180872 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:32.180811 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140de2de-7a2e-404c-b921-1d989fa29a49" path="/var/lib/kubelet/pods/140de2de-7a2e-404c-b921-1d989fa29a49/volumes" Nov 02 06:59:33.551547 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:33.551469 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 06:59:38.552263 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:59:38.552228 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 06:59:46.511519 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.511464 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" podUID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" containerName="main" containerID="cri-o://886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48" gracePeriod=45 Nov 02 06:59:46.778424 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.778371 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:59:46.874517 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874492 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.874656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874526 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.874656 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874566 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\" (UniqueName: \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.874780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874764 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.874834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874805 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.874834 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874761 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:46.874914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874773 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:46.874914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.874830 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\") pod \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\" (UID: \"ae24fec7-393e-45e1-b69f-8e8f504f3dec\") " Nov 02 06:59:46.875070 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.875053 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:46.875293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.875274 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:46.875293 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.875293 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:46.875437 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.875306 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:46.876864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.876821 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:46.876864 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.876853 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "kube-api-access-vwljl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:46.877021 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.876863 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm" pod "ae24fec7-393e-45e1-b69f-8e8f504f3dec" (UID: "ae24fec7-393e-45e1-b69f-8e8f504f3dec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:59:46.976914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.976887 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae24fec7-393e-45e1-b69f-8e8f504f3dec-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:46.976914 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.976911 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae24fec7-393e-45e1-b69f-8e8f504f3dec-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:46.977098 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:46.976920 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwljl\" (UniqueName: \"kubernetes.io/projected/ae24fec7-393e-45e1-b69f-8e8f504f3dec-kube-api-access-vwljl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:47.267539 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.267511 6273 generic.go:356] "Generic (PLEG): container finished" podID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" containerID="886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48" exitCode=0 Nov 02 06:59:47.267693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.267551 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" Nov 02 06:59:47.267693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.267601 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" event={"ID":"ae24fec7-393e-45e1-b69f-8e8f504f3dec","Type":"ContainerDied","Data":"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48"} Nov 02 06:59:47.267693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.267640 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7" event={"ID":"ae24fec7-393e-45e1-b69f-8e8f504f3dec","Type":"ContainerDied","Data":"788b8e31d26f478386c815684eb2b6904a3a48458e0a588914f9054e0e415f47"} Nov 02 06:59:47.267693 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.267660 6273 scope.go:122] "RemoveContainer" containerID="886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48" Nov 02 06:59:47.277780 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.277758 6273 scope.go:122] "RemoveContainer" containerID="886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48" Nov 02 06:59:47.278063 ip-10-0-142-103 kubenswrapper[6273]: E1102 06:59:47.278030 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48\": container with ID starting with 886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48 not found: ID does not exist" containerID="886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48" Nov 02 06:59:47.278144 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.278070 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48"} err="failed to get container status \"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48\": rpc error: code = NotFound desc = could not find container \"886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48\": container with ID starting with 886355e23c067bfffa3b70eb8c7d82e7efa213527a88141e4fd76134c4706f48 not found: ID does not exist" Nov 02 06:59:47.425317 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.425286 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:59:47.471800 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:47.471761 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8bc131/test-model-8bc131-kserve-569d4948cf-fcdp7"] Nov 02 06:59:48.178803 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:48.178776 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" path="/var/lib/kubelet/pods/ae24fec7-393e-45e1-b69f-8e8f504f3dec/volumes" Nov 02 06:59:59.497014 ip-10-0-142-103 kubenswrapper[6273]: I1102 06:59:59.496959 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 07:00:00.199805 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.199774 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:00.200326 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200310 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200326 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200341 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200346 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200359 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e2be5f02-a59c-42b4-93c1-a25cd85af574" containerName="maas-api" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200363 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2be5f02-a59c-42b4-93c1-a25cd85af574" containerName="maas-api" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200372 6273 cpu_manager.go:414] "containerMap: removing container" podUID="140de2de-7a2e-404c-b921-1d989fa29a49" containerName="main" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200377 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140de2de-7a2e-404c-b921-1d989fa29a49" containerName="main" Nov 02 07:00:00.200387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200387 6273 cpu_manager.go:414] "containerMap: removing container" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200391 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200399 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1c41e2f8-551c-4914-a518-b525edaa0b16" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200403 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41e2f8-551c-4914-a518-b525edaa0b16" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200415 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b2265b64-43b4-4e6a-b525-88b389e184f4" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200419 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2265b64-43b4-4e6a-b525-88b389e184f4" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200427 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200431 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200438 6273 cpu_manager.go:414] "containerMap: removing container" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200442 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200451 6273 cpu_manager.go:414] "containerMap: removing container" podUID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" containerName="main" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200455 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" containerName="main" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200473 6273 cpu_manager.go:414] "containerMap: removing container" podUID="967a17e3-bc9c-4528-b4c6-8ba1883905df" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200478 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967a17e3-bc9c-4528-b4c6-8ba1883905df" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200551 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="967a17e3-bc9c-4528-b4c6-8ba1883905df" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200561 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200568 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae3a01a-11a0-47e1-b147-5ced912bee0c" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200574 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2be5f02-a59c-42b4-93c1-a25cd85af574" containerName="maas-api" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200579 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="140de2de-7a2e-404c-b921-1d989fa29a49" containerName="main" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200586 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200592 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41e2f8-551c-4914-a518-b525edaa0b16" containerName="payload-pre-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200597 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2265b64-43b4-4e6a-b525-88b389e184f4" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200602 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6926d5-1b5b-452d-ad88-d6a6a5b26a3f" containerName="payload-processing" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200606 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e36a03-cf8c-487d-acea-5e9540769be8" containerName="istio-proxy" Nov 02 07:00:00.200640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.200611 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae24fec7-393e-45e1-b69f-8e8f504f3dec" containerName="main" Nov 02 07:00:00.203832 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.203810 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.303381 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.303339 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:00.312079 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.312050 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:00:00.327610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.327582 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g7p\" (UniqueName: \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.327737 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.327725 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.428755 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.428721 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.428964 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.428835 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g7p\" (UniqueName: \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.429425 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.429405 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.483503 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.483414 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g7p\" (UniqueName: \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.519417 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.519381 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:00.900430 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:00.900405 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:00.908716 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:00:00.908687 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593d48a2_7e21_441b_b578_02a7f1a6f251.slice/crio-d8899d02f03acb2ec4b90a2fef83eb5be871970f5e6c78479ec5642a635c93f4 WatchSource:0}: Error finding container d8899d02f03acb2ec4b90a2fef83eb5be871970f5e6c78479ec5642a635c93f4: Status 404 returned error can't find the container with id d8899d02f03acb2ec4b90a2fef83eb5be871970f5e6c78479ec5642a635c93f4 Nov 02 07:00:01.346305 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:01.346273 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerStarted","Data":"8b9adf014a11db153b1f59028f7dab1597a4126d5e9c80fb655bba60318287b5"} Nov 02 07:00:01.346465 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:01.346326 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerStarted","Data":"d8899d02f03acb2ec4b90a2fef83eb5be871970f5e6c78479ec5642a635c93f4"} Nov 02 07:00:01.452062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:01.451917 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" podStartSLOduration=1.451900802 podStartE2EDuration="1.451900802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:01.45182808 +0000 UTC m=+2555.425162766" watchObservedRunningTime="2026-11-02 07:00:01.451900802 +0000 UTC m=+2555.425235486" Nov 02 07:00:01.603025 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:01.602978 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 07:00:02.352984 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:02.352946 6273 generic.go:356] "Generic (PLEG): container finished" podID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerID="8b9adf014a11db153b1f59028f7dab1597a4126d5e9c80fb655bba60318287b5" exitCode=60 Nov 02 07:00:02.353183 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:02.353032 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerDied","Data":"8b9adf014a11db153b1f59028f7dab1597a4126d5e9c80fb655bba60318287b5"} Nov 02 07:00:02.353298 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:02.353285 6273 scope.go:122] "RemoveContainer" containerID="8b9adf014a11db153b1f59028f7dab1597a4126d5e9c80fb655bba60318287b5" Nov 02 07:00:03.360978 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:03.360947 6273 generic.go:356] "Generic (PLEG): container finished" podID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerID="ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27" exitCode=60 Nov 02 07:00:03.361440 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:03.361022 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerDied","Data":"ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27"} Nov 02 07:00:03.361440 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:03.361078 6273 scope.go:122] "RemoveContainer" containerID="8b9adf014a11db153b1f59028f7dab1597a4126d5e9c80fb655bba60318287b5" Nov 02 07:00:03.361440 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:03.361285 6273 scope.go:122] "RemoveContainer" containerID="ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27" Nov 02 07:00:03.361600 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:03.361534 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6_odh-ai-gateway-infra(593d48a2-7e21-441b-b578-02a7f1a6f251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" Nov 02 07:00:04.368979 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:04.368951 6273 scope.go:122] "RemoveContainer" containerID="ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27" Nov 02 07:00:04.369379 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:04.369204 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6_odh-ai-gateway-infra(593d48a2-7e21-441b-b578-02a7f1a6f251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" Nov 02 07:00:08.553033 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:08.552979 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:00:12.830857 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:12.830823 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 07:00:12.928635 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:12.928605 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 07:00:14.517583 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.517545 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" podUID="2eff15b3-d9b1-480c-b43e-795550a9895a" containerName="main" containerID="cri-o://08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707" gracePeriod=45 Nov 02 07:00:14.777836 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.777776 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 07:00:14.884742 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.884690 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\" (UniqueName: \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.884946 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.884773 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.884946 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.884815 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.884946 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.884871 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.885473 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.884969 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.885717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.885682 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\") pod \"2eff15b3-d9b1-480c-b43e-795550a9895a\" (UID: \"2eff15b3-d9b1-480c-b43e-795550a9895a\") " Nov 02 07:00:14.886388 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.886365 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:14.886725 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.886706 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:14.887009 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.886975 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:14.888086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.888061 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "kube-api-access-8np42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:14.889452 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.889425 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:14.892155 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.892123 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm" pod "2eff15b3-d9b1-480c-b43e-795550a9895a" (UID: "2eff15b3-d9b1-480c-b43e-795550a9895a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:14.987327 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987300 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8np42\" (UniqueName: \"kubernetes.io/projected/2eff15b3-d9b1-480c-b43e-795550a9895a-kube-api-access-8np42\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:14.987327 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987324 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:14.987497 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987333 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:14.987497 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987363 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:14.987497 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987371 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eff15b3-d9b1-480c-b43e-795550a9895a-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:14.987497 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:14.987381 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eff15b3-d9b1-480c-b43e-795550a9895a-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:15.433550 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.433515 6273 generic.go:356] "Generic (PLEG): container finished" podID="2eff15b3-d9b1-480c-b43e-795550a9895a" containerID="08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707" exitCode=0 Nov 02 07:00:15.433701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.433561 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" Nov 02 07:00:15.433701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.433600 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" event={"ID":"2eff15b3-d9b1-480c-b43e-795550a9895a","Type":"ContainerDied","Data":"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707"} Nov 02 07:00:15.433701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.433644 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m" event={"ID":"2eff15b3-d9b1-480c-b43e-795550a9895a","Type":"ContainerDied","Data":"3e58c785bc6e360743e2f6fe87031421cb231bd1ac17ee056e21ed60a25c8871"} Nov 02 07:00:15.433701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.433662 6273 scope.go:122] "RemoveContainer" containerID="08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707" Nov 02 07:00:15.443800 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.443783 6273 scope.go:122] "RemoveContainer" containerID="08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707" Nov 02 07:00:15.444064 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:15.444043 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707\": container with ID starting with 08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707 not found: ID does not exist" containerID="08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707" Nov 02 07:00:15.444139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.444070 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707"} err="failed to get container status \"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707\": rpc error: code = NotFound desc = could not find container \"08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707\": container with ID starting with 08039647e9d476f2e8560b4d1ddb2b1465df2ec24318b2fa467a6b7f3d5cc707 not found: ID does not exist" Nov 02 07:00:15.588038 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.588017 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 07:00:15.634449 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:15.634428 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5a85b0/test-model-5a85b0-kserve-6f484988fc-kb29m"] Nov 02 07:00:16.179264 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.179231 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eff15b3-d9b1-480c-b43e-795550a9895a" path="/var/lib/kubelet/pods/2eff15b3-d9b1-480c-b43e-795550a9895a/volumes" Nov 02 07:00:16.623752 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.623663 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" containerID="cri-o://4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6" gracePeriod=45 Nov 02 07:00:16.883304 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.883249 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 07:00:16.907078 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907055 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907208 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907091 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907208 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907140 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907280 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907244 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907329 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907275 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\" (UniqueName: \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907386 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907328 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\") pod \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\" (UID: \"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6\") " Nov 02 07:00:16.907449 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907329 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:16.907523 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907501 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:16.907577 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907519 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:16.907734 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907714 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:16.907734 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907733 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:16.907912 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.907744 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:16.909391 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.909368 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:16.909475 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.909386 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:16.909521 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:16.909464 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q" pod "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" (UID: "2db6bfd2-6e42-40ed-9fb3-bfa026d677c6"). InnerVolumeSpecName "kube-api-access-x4w5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.009109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.009078 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.009109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.009109 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w5q\" (UniqueName: \"kubernetes.io/projected/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-kube-api-access-x4w5q\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.009350 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.009124 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.447279 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.447247 6273 generic.go:356] "Generic (PLEG): container finished" podID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerID="4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6" exitCode=0 Nov 02 07:00:17.447279 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.447275 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" event={"ID":"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6","Type":"ContainerDied","Data":"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6"} Nov 02 07:00:17.447648 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.447299 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" Nov 02 07:00:17.447648 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.447314 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8" event={"ID":"2db6bfd2-6e42-40ed-9fb3-bfa026d677c6","Type":"ContainerDied","Data":"c66bc74478c5d39d06f691240f2602dabc4634c178854772b6730eeca898eceb"} Nov 02 07:00:17.447648 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.447333 6273 scope.go:122] "RemoveContainer" containerID="4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6" Nov 02 07:00:17.459246 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.459229 6273 scope.go:122] "RemoveContainer" containerID="4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6" Nov 02 07:00:17.459518 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:17.459498 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6\": container with ID starting with 4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6 not found: ID does not exist" containerID="4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6" Nov 02 07:00:17.459614 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.459522 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6"} err="failed to get container status \"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6\": rpc error: code = NotFound desc = could not find container \"4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6\": container with ID starting with 4af2abae90bba7e9d6cd17f3156f5df145b5a9a11261765f2b2b346fe518e9f6 not found: ID does not exist" Nov 02 07:00:17.600782 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.600719 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 07:00:17.647396 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.647364 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-581608/premium-w2-581608-kserve-c899d9d6-9lxz8"] Nov 02 07:00:17.832253 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.832180 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerName="payload-processing" containerID="cri-o://a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e" gracePeriod=25 Nov 02 07:00:17.855321 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.855298 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:00:17.928901 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:17.928865 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" containerID="cri-o://9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f" gracePeriod=25 Nov 02 07:00:18.006722 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:00:18.006691 6273 logging.go:55] [core] [Channel #5105 SubChannel #5107]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.136:50052", ServerName: "10.134.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.136:50052: connect: connection refused" Nov 02 07:00:18.008185 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.008147 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:18.128651 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.128619 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 07:00:18.173042 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.172980 6273 scope.go:122] "RemoveContainer" containerID="ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27" Nov 02 07:00:18.183558 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.183532 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" path="/var/lib/kubelet/pods/2db6bfd2-6e42-40ed-9fb3-bfa026d677c6/volumes" Nov 02 07:00:18.223495 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.223471 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\" (UniqueName: \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\") pod \"736b71f9-b690-4d11-9c9f-5bf91999bf49\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " Nov 02 07:00:18.223627 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.223559 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\") pod \"736b71f9-b690-4d11-9c9f-5bf91999bf49\" (UID: \"736b71f9-b690-4d11-9c9f-5bf91999bf49\") " Nov 02 07:00:18.223916 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.223891 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume" pod "736b71f9-b690-4d11-9c9f-5bf91999bf49" (UID: "736b71f9-b690-4d11-9c9f-5bf91999bf49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:18.225542 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.225517 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx" pod "736b71f9-b690-4d11-9c9f-5bf91999bf49" (UID: "736b71f9-b690-4d11-9c9f-5bf91999bf49"). InnerVolumeSpecName "kube-api-access-67vmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:18.324678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.324531 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736b71f9-b690-4d11-9c9f-5bf91999bf49-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:18.324678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.324557 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vmx\" (UniqueName: \"kubernetes.io/projected/736b71f9-b690-4d11-9c9f-5bf91999bf49-kube-api-access-67vmx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:18.454830 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.454800 6273 generic.go:356] "Generic (PLEG): container finished" podID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerID="a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e" exitCode=0 Nov 02 07:00:18.455026 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.454846 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" Nov 02 07:00:18.455026 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.454881 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" event={"ID":"736b71f9-b690-4d11-9c9f-5bf91999bf49","Type":"ContainerDied","Data":"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e"} Nov 02 07:00:18.455026 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.454935 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" event={"ID":"736b71f9-b690-4d11-9c9f-5bf91999bf49","Type":"ContainerDied","Data":"acd170f1a00a69e938a006d4616579d911795f90d691cbaa4a75b1731e8a2a84"} Nov 02 07:00:18.455026 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.454963 6273 scope.go:122] "RemoveContainer" containerID="a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e" Nov 02 07:00:18.464693 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.464677 6273 scope.go:122] "RemoveContainer" containerID="a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e" Nov 02 07:00:18.464974 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:18.464957 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e\": container with ID starting with a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e not found: ID does not exist" containerID="a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e" Nov 02 07:00:18.465127 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.465106 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e"} err="failed to get container status \"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e\": rpc error: code = NotFound desc = could not find container \"a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e\": container with ID starting with a4887c5523807cf453b9bc4b22f225999476dd431e23d2c8534b50b15fde4d8e not found: ID does not exist" Nov 02 07:00:18.608144 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.608111 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 07:00:18.611314 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.611281 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt"] Nov 02 07:00:18.992694 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:18.992660 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8bc131-5bd88d8ccc-mmcxt" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.136:50052\" within 1s: context deadline exceeded" Nov 02 07:00:19.464225 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:19.464196 6273 generic.go:356] "Generic (PLEG): container finished" podID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerID="44423b5f9f33ccc14ec26c976b14c8eec3e3640f8b93086ead863cab437d75be" exitCode=60 Nov 02 07:00:19.464406 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:19.464272 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerDied","Data":"44423b5f9f33ccc14ec26c976b14c8eec3e3640f8b93086ead863cab437d75be"} Nov 02 07:00:19.464406 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:19.464337 6273 scope.go:122] "RemoveContainer" containerID="ea446eaeb91f630c0f793220d97cf6963c630988866331b11a058ea822aa8e27" Nov 02 07:00:19.464676 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:19.464663 6273 scope.go:122] "RemoveContainer" containerID="44423b5f9f33ccc14ec26c976b14c8eec3e3640f8b93086ead863cab437d75be" Nov 02 07:00:19.464919 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:19.464903 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6_odh-ai-gateway-infra(593d48a2-7e21-441b-b578-02a7f1a6f251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" Nov 02 07:00:20.179758 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.179728 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" path="/var/lib/kubelet/pods/736b71f9-b690-4d11-9c9f-5bf91999bf49/volumes" Nov 02 07:00:20.584411 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.584326 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:20.726607 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.726582 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:20.752691 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.752668 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\") pod \"593d48a2-7e21-441b-b578-02a7f1a6f251\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " Nov 02 07:00:20.752853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.752821 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\" (UniqueName: \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\") pod \"593d48a2-7e21-441b-b578-02a7f1a6f251\" (UID: \"593d48a2-7e21-441b-b578-02a7f1a6f251\") " Nov 02 07:00:20.753093 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.753065 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca" pod "593d48a2-7e21-441b-b578-02a7f1a6f251" (UID: "593d48a2-7e21-441b-b578-02a7f1a6f251"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:20.753218 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.753204 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593d48a2-7e21-441b-b578-02a7f1a6f251-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:20.754779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.754757 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p" pod "593d48a2-7e21-441b-b578-02a7f1a6f251" (UID: "593d48a2-7e21-441b-b578-02a7f1a6f251"). InnerVolumeSpecName "kube-api-access-b7g7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:20.854651 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:20.854587 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g7p\" (UniqueName: \"kubernetes.io/projected/593d48a2-7e21-441b-b578-02a7f1a6f251-kube-api-access-b7g7p\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:21.479306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:21.479277 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" Nov 02 07:00:21.479707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:21.479280 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6" event={"ID":"593d48a2-7e21-441b-b578-02a7f1a6f251","Type":"ContainerDied","Data":"d8899d02f03acb2ec4b90a2fef83eb5be871970f5e6c78479ec5642a635c93f4"} Nov 02 07:00:21.479707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:21.479402 6273 scope.go:122] "RemoveContainer" containerID="44423b5f9f33ccc14ec26c976b14c8eec3e3640f8b93086ead863cab437d75be" Nov 02 07:00:21.634414 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:21.634385 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:21.640172 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:21.640146 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893380-t6fr6"] Nov 02 07:00:22.180058 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:22.180027 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" path="/var/lib/kubelet/pods/593d48a2-7e21-441b-b578-02a7f1a6f251/volumes" Nov 02 07:00:27.994139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:27.994106 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:37.993610 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:00:37.993580 6273 logging.go:55] [core] [Channel #5149 SubChannel #5150]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: connect: connection refused" Nov 02 07:00:38.211225 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.211161 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 07:00:38.325616 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.325584 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\" (UniqueName: \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\") pod \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " Nov 02 07:00:38.325794 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.325677 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\") pod \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\" (UID: \"5834064e-dac1-4f44-8fdb-be8b4dfd9224\") " Nov 02 07:00:38.326110 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.326084 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume" pod "5834064e-dac1-4f44-8fdb-be8b4dfd9224" (UID: "5834064e-dac1-4f44-8fdb-be8b4dfd9224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:38.328085 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.328051 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg" pod "5834064e-dac1-4f44-8fdb-be8b4dfd9224" (UID: "5834064e-dac1-4f44-8fdb-be8b4dfd9224"). InnerVolumeSpecName "kube-api-access-9thrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:38.426957 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.426929 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thrg\" (UniqueName: \"kubernetes.io/projected/5834064e-dac1-4f44-8fdb-be8b4dfd9224-kube-api-access-9thrg\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:38.426957 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.426954 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834064e-dac1-4f44-8fdb-be8b4dfd9224-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:38.552565 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:38.552492 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:00:38.580321 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.580290 6273 generic.go:356] "Generic (PLEG): container finished" podID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerID="9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f" exitCode=0 Nov 02 07:00:38.580460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.580329 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" Nov 02 07:00:38.580460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.580362 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" event={"ID":"5834064e-dac1-4f44-8fdb-be8b4dfd9224","Type":"ContainerDied","Data":"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f"} Nov 02 07:00:38.580460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.580418 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" event={"ID":"5834064e-dac1-4f44-8fdb-be8b4dfd9224","Type":"ContainerDied","Data":"bf47786bd8a321ebba0eb40a684868621dc07e9b07be8600d18231af2efbe828"} Nov 02 07:00:38.580460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.580439 6273 scope.go:122] "RemoveContainer" containerID="9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f" Nov 02 07:00:38.590748 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.590725 6273 scope.go:122] "RemoveContainer" containerID="9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f" Nov 02 07:00:38.591073 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:38.591051 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f\": container with ID starting with 9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f not found: ID does not exist" containerID="9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f" Nov 02 07:00:38.591157 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.591086 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f"} err="failed to get container status \"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f\": rpc error: code = NotFound desc = could not find container \"9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f\": container with ID starting with 9203d11d1ad26d52a9243629bb658ed73105b1f70f43a2a5d469a6f95efa197f not found: ID does not exist" Nov 02 07:00:38.734118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.734090 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 07:00:38.740889 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.740862 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx"] Nov 02 07:00:38.993630 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:38.993596 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8bc131-fc75d8c9f-wvwmx" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded" Nov 02 07:00:40.179576 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:40.179545 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" path="/var/lib/kubelet/pods/5834064e-dac1-4f44-8fdb-be8b4dfd9224/volumes" Nov 02 07:00:41.210078 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.210038 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 07:00:41.210495 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.210223 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" podUID="b3c50225-9ac4-4438-a308-6daf361948b5" containerName="maas-api" containerID="cri-o://e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb" gracePeriod=30 Nov 02 07:00:41.468672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.468615 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 07:00:41.561972 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.561942 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\") pod \"b3c50225-9ac4-4438-a308-6daf361948b5\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " Nov 02 07:00:41.562154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.562086 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\") pod \"b3c50225-9ac4-4438-a308-6daf361948b5\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " Nov 02 07:00:41.562154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.562128 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\" (UniqueName: \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\") pod \"b3c50225-9ac4-4438-a308-6daf361948b5\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " Nov 02 07:00:41.562306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.562288 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\") pod \"b3c50225-9ac4-4438-a308-6daf361948b5\" (UID: \"b3c50225-9ac4-4438-a308-6daf361948b5\") " Nov 02 07:00:41.562365 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.562325 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca" pod "b3c50225-9ac4-4438-a308-6daf361948b5" (UID: "b3c50225-9ac4-4438-a308-6daf361948b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:41.562773 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.562749 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c50225-9ac4-4438-a308-6daf361948b5-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:41.564403 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.564378 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs" pod "b3c50225-9ac4-4438-a308-6daf361948b5" (UID: "b3c50225-9ac4-4438-a308-6daf361948b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:41.564506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.564434 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls" pod "b3c50225-9ac4-4438-a308-6daf361948b5" (UID: "b3c50225-9ac4-4438-a308-6daf361948b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:41.564815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.564789 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78" pod "b3c50225-9ac4-4438-a308-6daf361948b5" (UID: "b3c50225-9ac4-4438-a308-6daf361948b5"). InnerVolumeSpecName "kube-api-access-mlj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:41.598351 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.598320 6273 generic.go:356] "Generic (PLEG): container finished" podID="b3c50225-9ac4-4438-a308-6daf361948b5" containerID="e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb" exitCode=0 Nov 02 07:00:41.598498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.598365 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" Nov 02 07:00:41.598498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.598394 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" event={"ID":"b3c50225-9ac4-4438-a308-6daf361948b5","Type":"ContainerDied","Data":"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb"} Nov 02 07:00:41.598498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.598439 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn" event={"ID":"b3c50225-9ac4-4438-a308-6daf361948b5","Type":"ContainerDied","Data":"e0a8d61d638cefe03681236daa93a3db008be374cd4092c221b13a1dd13df387"} Nov 02 07:00:41.598498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.598464 6273 scope.go:122] "RemoveContainer" containerID="e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb" Nov 02 07:00:41.610768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.610749 6273 scope.go:122] "RemoveContainer" containerID="e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb" Nov 02 07:00:41.611068 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:41.611043 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb\": container with ID starting with e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb not found: ID does not exist" containerID="e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb" Nov 02 07:00:41.611160 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.611084 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb"} err="failed to get container status \"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb\": rpc error: code = NotFound desc = could not find container \"e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb\": container with ID starting with e271145d9e4fc0498bd24d48fc03f5c612a31e3c9a7c2d3a95b58aa027c031cb not found: ID does not exist" Nov 02 07:00:41.663853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.663824 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:41.663853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.663848 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c50225-9ac4-4438-a308-6daf361948b5-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:41.663853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.663859 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlj78\" (UniqueName: \"kubernetes.io/projected/b3c50225-9ac4-4438-a308-6daf361948b5-kube-api-access-mlj78\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:41.751786 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.751678 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 07:00:41.753940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:41.753913 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ba11-74f8c8d5cb-dv2qn"] Nov 02 07:00:42.179498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.179471 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c50225-9ac4-4438-a308-6daf361948b5" path="/var/lib/kubelet/pods/b3c50225-9ac4-4438-a308-6daf361948b5/volumes" Nov 02 07:00:42.317928 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.317895 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 07:00:42.318388 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.318101 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerName="maas-api" containerID="cri-o://b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327" gracePeriod=30 Nov 02 07:00:42.589396 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.589375 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 07:00:42.604999 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.604959 6273 generic.go:356] "Generic (PLEG): container finished" podID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerID="b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327" exitCode=0 Nov 02 07:00:42.605118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.605027 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" Nov 02 07:00:42.605118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.605048 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" event={"ID":"b9bbee33-c01f-4d8e-8664-bb25a4534f44","Type":"ContainerDied","Data":"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327"} Nov 02 07:00:42.605118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.605100 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" event={"ID":"b9bbee33-c01f-4d8e-8664-bb25a4534f44","Type":"ContainerDied","Data":"8a4b508a81705451c7d69f606ea18cc61c8a6c607b08721eb721b84eb5f0221f"} Nov 02 07:00:42.605262 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.605122 6273 scope.go:122] "RemoveContainer" containerID="b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327" Nov 02 07:00:42.613827 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.613809 6273 scope.go:122] "RemoveContainer" containerID="b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327" Nov 02 07:00:42.614095 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:00:42.614078 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327\": container with ID starting with b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327 not found: ID does not exist" containerID="b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327" Nov 02 07:00:42.614168 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.614102 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327"} err="failed to get container status \"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327\": rpc error: code = NotFound desc = could not find container \"b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327\": container with ID starting with b208d0761cf2f338f42fd2b2e446fb0a6623da6fb0f6b40dad05482737c5a327 not found: ID does not exist" Nov 02 07:00:42.675330 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.675309 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") pod \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " Nov 02 07:00:42.675460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.675404 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\") pod \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " Nov 02 07:00:42.675460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.675440 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\") pod \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " Nov 02 07:00:42.675533 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.675490 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\") pod \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\" (UID: \"b9bbee33-c01f-4d8e-8664-bb25a4534f44\") " Nov 02 07:00:42.675903 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.675877 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca" pod "b9bbee33-c01f-4d8e-8664-bb25a4534f44" (UID: "b9bbee33-c01f-4d8e-8664-bb25a4534f44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:42.677530 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.677506 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml" pod "b9bbee33-c01f-4d8e-8664-bb25a4534f44" (UID: "b9bbee33-c01f-4d8e-8664-bb25a4534f44"). InnerVolumeSpecName "kube-api-access-tgwml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:42.677530 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.677515 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls" pod "b9bbee33-c01f-4d8e-8664-bb25a4534f44" (UID: "b9bbee33-c01f-4d8e-8664-bb25a4534f44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:42.677685 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.677533 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs" pod "b9bbee33-c01f-4d8e-8664-bb25a4534f44" (UID: "b9bbee33-c01f-4d8e-8664-bb25a4534f44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:42.776815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.776733 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:42.776815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.776762 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwml\" (UniqueName: \"kubernetes.io/projected/b9bbee33-c01f-4d8e-8664-bb25a4534f44-kube-api-access-tgwml\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:42.776815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.776772 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbee33-c01f-4d8e-8664-bb25a4534f44-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:42.776815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:42.776781 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bbee33-c01f-4d8e-8664-bb25a4534f44-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:43.018583 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:43.018549 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 07:00:43.024708 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:43.024682 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb"] Nov 02 07:00:44.178784 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:44.178749 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" path="/var/lib/kubelet/pods/b9bbee33-c01f-4d8e-8664-bb25a4534f44/volumes" Nov 02 07:00:45.575844 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:45.575803 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8bc131-dc965cd4b-9gzbb" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.139:8443/health\": context deadline exceeded" Nov 02 07:00:49.934553 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:00:49.934516 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:01:02.087839 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:02.087804 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 07:01:02.189176 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:02.189143 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 07:01:04.384220 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:04.384150 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 07:01:04.484774 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:04.484743 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 07:01:07.088178 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:07.088138 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" containerID="cri-o://2003cb2c867a25c6b969d8b208ca1832dcb08d84f801436b4400f6fcb1497006" gracePeriod=25 Nov 02 07:01:07.189654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:07.189614 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" containerID="cri-o://f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c" gracePeriod=25 Nov 02 07:01:08.552185 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:08.552149 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:01:09.385218 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.385174 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" podUID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" containerName="payload-processing" containerID="cri-o://4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99" gracePeriod=25 Nov 02 07:01:09.485608 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.485559 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" containerID="cri-o://95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73" gracePeriod=25 Nov 02 07:01:09.647747 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.647691 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 07:01:09.747156 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.747130 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\") pod \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " Nov 02 07:01:09.747322 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.747216 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\" (UniqueName: \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\") pod \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\" (UID: \"fa4b46a8-1510-497f-a0fa-043f3b6b7c67\") " Nov 02 07:01:09.747605 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.747576 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume" pod "fa4b46a8-1510-497f-a0fa-043f3b6b7c67" (UID: "fa4b46a8-1510-497f-a0fa-043f3b6b7c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:09.749289 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.749268 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4" pod "fa4b46a8-1510-497f-a0fa-043f3b6b7c67" (UID: "fa4b46a8-1510-497f-a0fa-043f3b6b7c67"). InnerVolumeSpecName "kube-api-access-2jmf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:09.765315 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.765283 6273 generic.go:356] "Generic (PLEG): container finished" podID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" containerID="4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99" exitCode=0 Nov 02 07:01:09.765431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.765322 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" Nov 02 07:01:09.765431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.765363 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" event={"ID":"fa4b46a8-1510-497f-a0fa-043f3b6b7c67","Type":"ContainerDied","Data":"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99"} Nov 02 07:01:09.765431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.765421 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch" event={"ID":"fa4b46a8-1510-497f-a0fa-043f3b6b7c67","Type":"ContainerDied","Data":"10648f3ce7757765ff011a1d063777b16ffdd6584362bd3323488b2210937f07"} Nov 02 07:01:09.765556 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.765449 6273 scope.go:122] "RemoveContainer" containerID="4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99" Nov 02 07:01:09.777029 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.777012 6273 scope.go:122] "RemoveContainer" containerID="4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99" Nov 02 07:01:09.777283 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:09.777266 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99\": container with ID starting with 4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99 not found: ID does not exist" containerID="4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99" Nov 02 07:01:09.777393 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.777293 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99"} err="failed to get container status \"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99\": rpc error: code = NotFound desc = could not find container \"4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99\": container with ID starting with 4bb0bd668962c982ada666453c6740f4698298c8a277de9e88a6bda422aede99 not found: ID does not exist" Nov 02 07:01:09.848749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.848725 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jmf4\" (UniqueName: \"kubernetes.io/projected/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-kube-api-access-2jmf4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:09.848749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.848748 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4b46a8-1510-497f-a0fa-043f3b6b7c67-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:09.923410 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.923291 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 07:01:09.925533 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:09.925511 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e9f81-5865cc48c7-zvxch"] Nov 02 07:01:10.179772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:10.179706 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" path="/var/lib/kubelet/pods/fa4b46a8-1510-497f-a0fa-043f3b6b7c67/volumes" Nov 02 07:01:10.689358 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:10.689324 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:10.689358 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:10.689346 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:14.313507 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:14.313469 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:19.411620 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.411577 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 07:01:19.412107 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.411846 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" podUID="e8a735c0-bc8e-4704-84b6-add16df8fecd" containerName="maas-api" containerID="cri-o://d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5" gracePeriod=30 Nov 02 07:01:19.674756 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.674700 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 07:01:19.747899 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.747871 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\") pod \"e8a735c0-bc8e-4704-84b6-add16df8fecd\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " Nov 02 07:01:19.748063 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.748041 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\" (UniqueName: \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\") pod \"e8a735c0-bc8e-4704-84b6-add16df8fecd\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " Nov 02 07:01:19.748124 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.748080 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\") pod \"e8a735c0-bc8e-4704-84b6-add16df8fecd\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " Nov 02 07:01:19.748171 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.748143 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") pod \"e8a735c0-bc8e-4704-84b6-add16df8fecd\" (UID: \"e8a735c0-bc8e-4704-84b6-add16df8fecd\") " Nov 02 07:01:19.748477 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.748451 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca" pod "e8a735c0-bc8e-4704-84b6-add16df8fecd" (UID: "e8a735c0-bc8e-4704-84b6-add16df8fecd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:19.750179 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.750153 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh" pod "e8a735c0-bc8e-4704-84b6-add16df8fecd" (UID: "e8a735c0-bc8e-4704-84b6-add16df8fecd"). InnerVolumeSpecName "kube-api-access-c7wfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:19.750278 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.750183 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs" pod "e8a735c0-bc8e-4704-84b6-add16df8fecd" (UID: "e8a735c0-bc8e-4704-84b6-add16df8fecd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:19.750278 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.750200 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls" pod "e8a735c0-bc8e-4704-84b6-add16df8fecd" (UID: "e8a735c0-bc8e-4704-84b6-add16df8fecd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:19.820812 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.820765 6273 generic.go:356] "Generic (PLEG): container finished" podID="e8a735c0-bc8e-4704-84b6-add16df8fecd" containerID="d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5" exitCode=0 Nov 02 07:01:19.820812 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.820813 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" Nov 02 07:01:19.821070 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.820860 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" event={"ID":"e8a735c0-bc8e-4704-84b6-add16df8fecd","Type":"ContainerDied","Data":"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5"} Nov 02 07:01:19.821070 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.820917 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6" event={"ID":"e8a735c0-bc8e-4704-84b6-add16df8fecd","Type":"ContainerDied","Data":"cb65e53ae6fee008954e0eecba02460f29059d21e3d18d44184bd486ee1d86bc"} Nov 02 07:01:19.821070 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.820938 6273 scope.go:122] "RemoveContainer" containerID="d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5" Nov 02 07:01:19.831395 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.831378 6273 scope.go:122] "RemoveContainer" containerID="d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5" Nov 02 07:01:19.831649 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:19.831632 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5\": container with ID starting with d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5 not found: ID does not exist" containerID="d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5" Nov 02 07:01:19.831701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.831657 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5"} err="failed to get container status \"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5\": rpc error: code = NotFound desc = could not find container \"d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5\": container with ID starting with d355c904831ba776425d5c085e412b1acf51e6e11ee2a07b9b158f0ae54a00d5 not found: ID does not exist" Nov 02 07:01:19.849325 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.849303 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:19.849325 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.849326 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a735c0-bc8e-4704-84b6-add16df8fecd-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:19.849443 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.849336 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7wfh\" (UniqueName: \"kubernetes.io/projected/e8a735c0-bc8e-4704-84b6-add16df8fecd-kube-api-access-c7wfh\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:19.849443 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.849346 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a735c0-bc8e-4704-84b6-add16df8fecd-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:19.974820 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.974731 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 07:01:19.976230 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:19.976207 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a85b0-9f98b865b-hrkv6"] Nov 02 07:01:20.181009 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.180955 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a735c0-bc8e-4704-84b6-add16df8fecd" path="/var/lib/kubelet/pods/e8a735c0-bc8e-4704-84b6-add16df8fecd/volumes" Nov 02 07:01:20.545859 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.545826 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 07:01:20.546286 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.546028 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" podUID="8e652264-bcbd-45f3-9a58-0c42115f5529" containerName="maas-api" containerID="cri-o://2534aa1bc4bbb4ef4d039fdd84889b71429c0b3609e9f05f2eab509cbeed6fe6" gracePeriod=30 Nov 02 07:01:20.690209 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.690166 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:20.690382 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.690293 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:20.828688 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.828658 6273 generic.go:356] "Generic (PLEG): container finished" podID="8e652264-bcbd-45f3-9a58-0c42115f5529" containerID="2534aa1bc4bbb4ef4d039fdd84889b71429c0b3609e9f05f2eab509cbeed6fe6" exitCode=0 Nov 02 07:01:20.828850 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.828726 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" event={"ID":"8e652264-bcbd-45f3-9a58-0c42115f5529","Type":"ContainerDied","Data":"2534aa1bc4bbb4ef4d039fdd84889b71429c0b3609e9f05f2eab509cbeed6fe6"} Nov 02 07:01:20.828850 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.828789 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" event={"ID":"8e652264-bcbd-45f3-9a58-0c42115f5529","Type":"ContainerDied","Data":"6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc"} Nov 02 07:01:20.828850 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.828817 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f49aba9f81ee3ca100fd7c3a15700be20f4dabbd30c2c9ab8aa92de2d8c76cc" Nov 02 07:01:20.838472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.838450 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 07:01:20.965556 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.965522 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\") pod \"8e652264-bcbd-45f3-9a58-0c42115f5529\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " Nov 02 07:01:20.965858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.965581 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\" (UniqueName: \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\") pod \"8e652264-bcbd-45f3-9a58-0c42115f5529\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " Nov 02 07:01:20.965858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.965711 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\") pod \"8e652264-bcbd-45f3-9a58-0c42115f5529\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " Nov 02 07:01:20.965858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.965810 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\") pod \"8e652264-bcbd-45f3-9a58-0c42115f5529\" (UID: \"8e652264-bcbd-45f3-9a58-0c42115f5529\") " Nov 02 07:01:20.966392 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.966362 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca" pod "8e652264-bcbd-45f3-9a58-0c42115f5529" (UID: "8e652264-bcbd-45f3-9a58-0c42115f5529"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:20.968266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.968229 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88" pod "8e652264-bcbd-45f3-9a58-0c42115f5529" (UID: "8e652264-bcbd-45f3-9a58-0c42115f5529"). InnerVolumeSpecName "kube-api-access-2zk88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:20.968266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.968249 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs" pod "8e652264-bcbd-45f3-9a58-0c42115f5529" (UID: "8e652264-bcbd-45f3-9a58-0c42115f5529"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:20.968605 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:20.968547 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls" pod "8e652264-bcbd-45f3-9a58-0c42115f5529" (UID: "8e652264-bcbd-45f3-9a58-0c42115f5529"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:21.067538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.067459 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:21.067538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.067488 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e652264-bcbd-45f3-9a58-0c42115f5529-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:21.067538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.067498 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zk88\" (UniqueName: \"kubernetes.io/projected/8e652264-bcbd-45f3-9a58-0c42115f5529-kube-api-access-2zk88\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:21.067538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.067507 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e652264-bcbd-45f3-9a58-0c42115f5529-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:21.834623 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.834594 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt" Nov 02 07:01:21.987728 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:21.987695 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 07:01:22.034090 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:22.034065 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e9f81-6858cd87b6-dhpxt"] Nov 02 07:01:22.179341 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:22.179312 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e652264-bcbd-45f3-9a58-0c42115f5529" path="/var/lib/kubelet/pods/8e652264-bcbd-45f3-9a58-0c42115f5529/volumes" Nov 02 07:01:24.312869 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:24.312814 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:01:27.339507 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.339478 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 07:01:27.367404 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.367375 6273 scope.go:122] "RemoveContainer" containerID="2003cb2c867a25c6b969d8b208ca1832dcb08d84f801436b4400f6fcb1497006" Nov 02 07:01:27.379038 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.379017 6273 scope.go:122] "RemoveContainer" containerID="2534aa1bc4bbb4ef4d039fdd84889b71429c0b3609e9f05f2eab509cbeed6fe6" Nov 02 07:01:27.433895 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.433872 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\") pod \"b2dd1a73-8743-4298-b812-eb823cede79d\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " Nov 02 07:01:27.434013 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.433931 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\" (UniqueName: \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\") pod \"b2dd1a73-8743-4298-b812-eb823cede79d\" (UID: \"b2dd1a73-8743-4298-b812-eb823cede79d\") " Nov 02 07:01:27.434283 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.434259 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume" pod "b2dd1a73-8743-4298-b812-eb823cede79d" (UID: "b2dd1a73-8743-4298-b812-eb823cede79d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:27.435801 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.435783 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl" pod "b2dd1a73-8743-4298-b812-eb823cede79d" (UID: "b2dd1a73-8743-4298-b812-eb823cede79d"). InnerVolumeSpecName "kube-api-access-tqbnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:27.448903 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.448880 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 07:01:27.534785 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.534753 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\") pod \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " Nov 02 07:01:27.534903 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.534805 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\" (UniqueName: \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\") pod \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\" (UID: \"6d438b90-6711-4ce0-bb13-5c3684e8b4b2\") " Nov 02 07:01:27.535126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.535112 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd1a73-8743-4298-b812-eb823cede79d-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:27.535169 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.535128 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbnl\" (UniqueName: \"kubernetes.io/projected/b2dd1a73-8743-4298-b812-eb823cede79d-kube-api-access-tqbnl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:27.535203 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.535164 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume" pod "6d438b90-6711-4ce0-bb13-5c3684e8b4b2" (UID: "6d438b90-6711-4ce0-bb13-5c3684e8b4b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:27.536742 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.536726 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km" pod "6d438b90-6711-4ce0-bb13-5c3684e8b4b2" (UID: "6d438b90-6711-4ce0-bb13-5c3684e8b4b2"). InnerVolumeSpecName "kube-api-access-hz7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:27.636521 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.636445 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:27.636521 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.636473 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7km\" (UniqueName: \"kubernetes.io/projected/6d438b90-6711-4ce0-bb13-5c3684e8b4b2-kube-api-access-hz7km\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:27.684042 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.683976 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:01:27.871732 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871701 6273 generic.go:356] "Generic (PLEG): container finished" podID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerID="f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c" exitCode=0 Nov 02 07:01:27.871914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871746 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" Nov 02 07:01:27.871914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871790 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" event={"ID":"6d438b90-6711-4ce0-bb13-5c3684e8b4b2","Type":"ContainerDied","Data":"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c"} Nov 02 07:01:27.871914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871844 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc" event={"ID":"6d438b90-6711-4ce0-bb13-5c3684e8b4b2","Type":"ContainerDied","Data":"e74719556b95dd0b4ffe4d46b29159c23f4f6cc34942e858a32ea26168cce639"} Nov 02 07:01:27.871914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871870 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" event={"ID":"b2dd1a73-8743-4298-b812-eb823cede79d","Type":"ContainerDied","Data":"2003cb2c867a25c6b969d8b208ca1832dcb08d84f801436b4400f6fcb1497006"} Nov 02 07:01:27.871914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871893 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" event={"ID":"b2dd1a73-8743-4298-b812-eb823cede79d","Type":"ContainerDied","Data":"efe9605abd9125d93862e36dcd746fd86f898a031fd8fd143e32a266a8d8f87b"} Nov 02 07:01:27.872199 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871919 6273 scope.go:122] "RemoveContainer" containerID="f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c" Nov 02 07:01:27.872199 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.871960 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx" Nov 02 07:01:27.882283 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.882266 6273 scope.go:122] "RemoveContainer" containerID="f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c" Nov 02 07:01:27.882540 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:27.882520 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c\": container with ID starting with f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c not found: ID does not exist" containerID="f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c" Nov 02 07:01:27.882584 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:27.882549 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c"} err="failed to get container status \"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c\": rpc error: code = NotFound desc = could not find container \"f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c\": container with ID starting with f1f1f0db47908e4a18f3c68200e462b501681d62f4a873cc05137a721e62d04c not found: ID does not exist" Nov 02 07:01:28.092355 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:28.089599 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 07:01:28.092355 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:28.092306 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a85b0-69886c67d5-w8kbx"] Nov 02 07:01:28.179039 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:28.179008 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" path="/var/lib/kubelet/pods/b2dd1a73-8743-4298-b812-eb823cede79d/volumes" Nov 02 07:01:28.196782 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:28.196751 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 07:01:28.200235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:28.200206 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a85b0-5575c4c999-mhjtc"] Nov 02 07:01:29.765075 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.765048 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 07:01:29.859790 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.859747 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\") pod \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " Nov 02 07:01:29.859942 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.859895 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\" (UniqueName: \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\") pod \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\" (UID: \"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb\") " Nov 02 07:01:29.860415 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.860310 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume" pod "59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" (UID: "59e60dab-0aaa-4b85-8a50-9b031f9b0fcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:29.866017 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.862510 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx" pod "59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" (UID: "59e60dab-0aaa-4b85-8a50-9b031f9b0fcb"). InnerVolumeSpecName "kube-api-access-86htx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:29.890672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.890629 6273 generic.go:356] "Generic (PLEG): container finished" podID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerID="95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73" exitCode=0 Nov 02 07:01:29.890855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.890783 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" Nov 02 07:01:29.892120 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.892068 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" event={"ID":"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb","Type":"ContainerDied","Data":"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73"} Nov 02 07:01:29.892757 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.892731 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5" event={"ID":"59e60dab-0aaa-4b85-8a50-9b031f9b0fcb","Type":"ContainerDied","Data":"581799d676394fcda04e403e66bf6c3e64a3d61a9409628f15086ed6214de9dc"} Nov 02 07:01:29.892870 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.892776 6273 scope.go:122] "RemoveContainer" containerID="95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73" Nov 02 07:01:29.911115 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.911090 6273 scope.go:122] "RemoveContainer" containerID="95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73" Nov 02 07:01:29.911394 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:29.911366 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73\": container with ID starting with 95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73 not found: ID does not exist" containerID="95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73" Nov 02 07:01:29.911445 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.911405 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73"} err="failed to get container status \"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73\": rpc error: code = NotFound desc = could not find container \"95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73\": container with ID starting with 95341af2f0249093dfa50cf2c670e68b043fbb29af36e07f10cab4022db88e73 not found: ID does not exist" Nov 02 07:01:29.961080 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.961054 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:29.961080 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:29.961078 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86htx\" (UniqueName: \"kubernetes.io/projected/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb-kube-api-access-86htx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:30.049126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:30.049061 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 07:01:30.059114 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:30.059089 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e9f81-7ff46d579f-jpnr5"] Nov 02 07:01:30.178973 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:30.178941 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" path="/var/lib/kubelet/pods/59e60dab-0aaa-4b85-8a50-9b031f9b0fcb/volumes" Nov 02 07:01:30.179341 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:30.179327 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" path="/var/lib/kubelet/pods/6d438b90-6711-4ce0-bb13-5c3684e8b4b2/volumes" Nov 02 07:01:33.977375 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:33.977345 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 07:01:38.551565 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:38.551530 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:01:48.997095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:48.997052 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" containerID="cri-o://929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6" gracePeriod=45 Nov 02 07:01:49.255263 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.255209 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 07:01:49.353086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353059 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353243 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353148 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353243 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353175 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353243 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353242 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353289 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\" (UniqueName: \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353338 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\") pod \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\" (UID: \"c1146e3e-9db0-40fb-8b6a-99d8398524e4\") " Nov 02 07:01:49.353429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353365 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.353604 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353422 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.353604 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353506 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.353757 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353733 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.353822 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353783 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.353822 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.353799 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.355815 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.355760 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "kube-api-access-nfh2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:49.356088 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.356063 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.357355 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.357332 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs" pod "c1146e3e-9db0-40fb-8b6a-99d8398524e4" (UID: "c1146e3e-9db0-40fb-8b6a-99d8398524e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:49.454692 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.454659 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfh2k\" (UniqueName: \"kubernetes.io/projected/c1146e3e-9db0-40fb-8b6a-99d8398524e4-kube-api-access-nfh2k\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.454692 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.454695 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1146e3e-9db0-40fb-8b6a-99d8398524e4-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.454881 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.454709 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1146e3e-9db0-40fb-8b6a-99d8398524e4-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.990075 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.990035 6273 generic.go:356] "Generic (PLEG): container finished" podID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerID="929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6" exitCode=0 Nov 02 07:01:49.990284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.990080 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" Nov 02 07:01:49.990284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.990125 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" event={"ID":"c1146e3e-9db0-40fb-8b6a-99d8398524e4","Type":"ContainerDied","Data":"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6"} Nov 02 07:01:49.990284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.990170 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h" event={"ID":"c1146e3e-9db0-40fb-8b6a-99d8398524e4","Type":"ContainerDied","Data":"c880b5bcf699eb4ee344b78493bad98c3d3d5bad99f90f1f8354c37dab5e5006"} Nov 02 07:01:49.990284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.990190 6273 scope.go:122] "RemoveContainer" containerID="929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6" Nov 02 07:01:50.000060 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:49.999901 6273 scope.go:122] "RemoveContainer" containerID="929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6" Nov 02 07:01:50.000328 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:01:50.000219 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6\": container with ID starting with 929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6 not found: ID does not exist" containerID="929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6" Nov 02 07:01:50.000328 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:50.000246 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6"} err="failed to get container status \"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6\": rpc error: code = NotFound desc = could not find container \"929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6\": container with ID starting with 929b21b463616862ded975711a80317323c8ce2191cf1a0e4ed8c78894e8fda6 not found: ID does not exist" Nov 02 07:01:50.146667 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:50.146640 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 07:01:50.150506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:50.150482 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-w1-cdfce9-kserve-85d8bc4f8b-kkk6h"] Nov 02 07:01:50.178262 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:50.178241 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" path="/var/lib/kubelet/pods/c1146e3e-9db0-40fb-8b6a-99d8398524e4/volumes" Nov 02 07:01:58.788610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:58.788577 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:01:59.550650 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.550617 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 07:01:59.550832 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.550816 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" podUID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" containerName="maas-api" containerID="cri-o://f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc" gracePeriod=30 Nov 02 07:01:59.802733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.802680 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 07:01:59.963644 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.963609 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\" (UniqueName: \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\") pod \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " Nov 02 07:01:59.963850 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.963673 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") pod \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " Nov 02 07:01:59.963850 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.963750 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\") pod \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " Nov 02 07:01:59.964047 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.963902 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\") pod \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\" (UID: \"ed43483b-809e-4d2c-8fbc-e89f44d691b0\") " Nov 02 07:01:59.964398 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.964369 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca" pod "ed43483b-809e-4d2c-8fbc-e89f44d691b0" (UID: "ed43483b-809e-4d2c-8fbc-e89f44d691b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:59.964519 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.964491 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed43483b-809e-4d2c-8fbc-e89f44d691b0-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:59.965833 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.965807 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls" pod "ed43483b-809e-4d2c-8fbc-e89f44d691b0" (UID: "ed43483b-809e-4d2c-8fbc-e89f44d691b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:59.965951 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.965855 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf" pod "ed43483b-809e-4d2c-8fbc-e89f44d691b0" (UID: "ed43483b-809e-4d2c-8fbc-e89f44d691b0"). InnerVolumeSpecName "kube-api-access-thnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:59.965951 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:01:59.965877 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs" pod "ed43483b-809e-4d2c-8fbc-e89f44d691b0" (UID: "ed43483b-809e-4d2c-8fbc-e89f44d691b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:00.044111 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.044077 6273 generic.go:356] "Generic (PLEG): container finished" podID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" containerID="f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc" exitCode=0 Nov 02 07:02:00.044297 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.044126 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" Nov 02 07:02:00.044297 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.044162 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" event={"ID":"ed43483b-809e-4d2c-8fbc-e89f44d691b0","Type":"ContainerDied","Data":"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc"} Nov 02 07:02:00.044297 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.044233 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5" event={"ID":"ed43483b-809e-4d2c-8fbc-e89f44d691b0","Type":"ContainerDied","Data":"0f4acf8dd5e71a8ebe9832105872faaa802b89a1473b241d530038447fc06172"} Nov 02 07:02:00.044297 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.044263 6273 scope.go:122] "RemoveContainer" containerID="f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc" Nov 02 07:02:00.054256 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.054237 6273 scope.go:122] "RemoveContainer" containerID="f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc" Nov 02 07:02:00.054529 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:00.054510 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc\": container with ID starting with f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc not found: ID does not exist" containerID="f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc" Nov 02 07:02:00.054580 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.054537 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc"} err="failed to get container status \"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc\": rpc error: code = NotFound desc = could not find container \"f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc\": container with ID starting with f730492c15862c4194d1be6489db3aa22d6bd5aa26d2917d28e0a403f2e317cc not found: ID does not exist" Nov 02 07:02:00.065115 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.065093 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thnnf\" (UniqueName: \"kubernetes.io/projected/ed43483b-809e-4d2c-8fbc-e89f44d691b0-kube-api-access-thnnf\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:00.065115 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.065113 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:00.065219 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.065122 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed43483b-809e-4d2c-8fbc-e89f44d691b0-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:00.198774 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.198738 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 07:02:00.203315 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:00.203266 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e96be-86d65895c6-g95v5"] Nov 02 07:02:02.178226 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:02.178189 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" path="/var/lib/kubelet/pods/ed43483b-809e-4d2c-8fbc-e89f44d691b0/volumes" Nov 02 07:02:08.555959 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:08.555906 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:02:20.477437 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:20.477402 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 07:02:20.576856 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:20.576815 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 07:02:20.921818 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:20.921788 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 07:02:20.922037 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:20.921976 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" podUID="03c60f2f-384b-46b2-8c57-38c03f872876" containerName="maas-api" containerID="cri-o://a7003fe3d2beb1a1cff783aad45ac129214cd078c84d6d2c8e4feaa36a833476" gracePeriod=30 Nov 02 07:02:21.155779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.155743 6273 generic.go:356] "Generic (PLEG): container finished" podID="03c60f2f-384b-46b2-8c57-38c03f872876" containerID="a7003fe3d2beb1a1cff783aad45ac129214cd078c84d6d2c8e4feaa36a833476" exitCode=0 Nov 02 07:02:21.155917 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.155812 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" event={"ID":"03c60f2f-384b-46b2-8c57-38c03f872876","Type":"ContainerDied","Data":"a7003fe3d2beb1a1cff783aad45ac129214cd078c84d6d2c8e4feaa36a833476"} Nov 02 07:02:21.178457 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.178403 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 07:02:21.279569 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.279534 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") pod \"03c60f2f-384b-46b2-8c57-38c03f872876\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " Nov 02 07:02:21.279775 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.279608 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\") pod \"03c60f2f-384b-46b2-8c57-38c03f872876\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " Nov 02 07:02:21.279775 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.279699 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\" (UniqueName: \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\") pod \"03c60f2f-384b-46b2-8c57-38c03f872876\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " Nov 02 07:02:21.279904 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.279816 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\") pod \"03c60f2f-384b-46b2-8c57-38c03f872876\" (UID: \"03c60f2f-384b-46b2-8c57-38c03f872876\") " Nov 02 07:02:21.280044 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.279972 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca" pod "03c60f2f-384b-46b2-8c57-38c03f872876" (UID: "03c60f2f-384b-46b2-8c57-38c03f872876"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:21.280345 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.280321 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c60f2f-384b-46b2-8c57-38c03f872876-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:21.281852 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.281829 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd" pod "03c60f2f-384b-46b2-8c57-38c03f872876" (UID: "03c60f2f-384b-46b2-8c57-38c03f872876"). InnerVolumeSpecName "kube-api-access-zl9rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:21.281934 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.281884 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls" pod "03c60f2f-384b-46b2-8c57-38c03f872876" (UID: "03c60f2f-384b-46b2-8c57-38c03f872876"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:21.281934 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.281903 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs" pod "03c60f2f-384b-46b2-8c57-38c03f872876" (UID: "03c60f2f-384b-46b2-8c57-38c03f872876"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:21.381465 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.381430 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:21.381465 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.381463 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c60f2f-384b-46b2-8c57-38c03f872876-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:21.381674 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:21.381476 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9rd\" (UniqueName: \"kubernetes.io/projected/03c60f2f-384b-46b2-8c57-38c03f872876-kube-api-access-zl9rd\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:22.162982 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:22.162947 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" Nov 02 07:02:22.163445 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:22.162974 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t" event={"ID":"03c60f2f-384b-46b2-8c57-38c03f872876","Type":"ContainerDied","Data":"04ed84418ecc9b22ac2d12f21991412232551a280b8c1f1bd5c66b644065725e"} Nov 02 07:02:22.163445 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:22.163063 6273 scope.go:122] "RemoveContainer" containerID="a7003fe3d2beb1a1cff783aad45ac129214cd078c84d6d2c8e4feaa36a833476" Nov 02 07:02:22.317184 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:22.317147 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 07:02:22.363933 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:22.363896 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-6e2657-7ddd85d686-zkt6t"] Nov 02 07:02:24.179876 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:24.179840 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c60f2f-384b-46b2-8c57-38c03f872876" path="/var/lib/kubelet/pods/03c60f2f-384b-46b2-8c57-38c03f872876/volumes" Nov 02 07:02:25.478608 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.478542 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" podUID="842d153e-144e-4f7f-ad3c-03360e8e2403" containerName="payload-processing" containerID="cri-o://8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b" gracePeriod=25 Nov 02 07:02:25.577874 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.577832 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" podUID="4fe0da50-df40-4fc7-a387-8734f87c2080" containerName="payload-pre-processing" containerID="cri-o://13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4" gracePeriod=25 Nov 02 07:02:25.744006 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.743926 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 07:02:25.782402 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.781973 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:02:25.829766 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.829729 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\") pod \"842d153e-144e-4f7f-ad3c-03360e8e2403\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " Nov 02 07:02:25.829956 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.829875 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\" (UniqueName: \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\") pod \"842d153e-144e-4f7f-ad3c-03360e8e2403\" (UID: \"842d153e-144e-4f7f-ad3c-03360e8e2403\") " Nov 02 07:02:25.830239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.830212 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume" pod "842d153e-144e-4f7f-ad3c-03360e8e2403" (UID: "842d153e-144e-4f7f-ad3c-03360e8e2403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:25.830338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.830321 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842d153e-144e-4f7f-ad3c-03360e8e2403-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:25.832212 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.832184 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz" pod "842d153e-144e-4f7f-ad3c-03360e8e2403" (UID: "842d153e-144e-4f7f-ad3c-03360e8e2403"). InnerVolumeSpecName "kube-api-access-hvprz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:25.849434 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.849409 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 07:02:25.931629 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.931594 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\" (UniqueName: \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\") pod \"4fe0da50-df40-4fc7-a387-8734f87c2080\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " Nov 02 07:02:25.932014 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.931720 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\") pod \"4fe0da50-df40-4fc7-a387-8734f87c2080\" (UID: \"4fe0da50-df40-4fc7-a387-8734f87c2080\") " Nov 02 07:02:25.932190 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.932160 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvprz\" (UniqueName: \"kubernetes.io/projected/842d153e-144e-4f7f-ad3c-03360e8e2403-kube-api-access-hvprz\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:25.933358 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.933324 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume" pod "4fe0da50-df40-4fc7-a387-8734f87c2080" (UID: "4fe0da50-df40-4fc7-a387-8734f87c2080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:25.935059 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:25.935034 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn" pod "4fe0da50-df40-4fc7-a387-8734f87c2080" (UID: "4fe0da50-df40-4fc7-a387-8734f87c2080"). InnerVolumeSpecName "kube-api-access-nktgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:26.033522 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.033436 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktgn\" (UniqueName: \"kubernetes.io/projected/4fe0da50-df40-4fc7-a387-8734f87c2080-kube-api-access-nktgn\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:26.033522 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.033469 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe0da50-df40-4fc7-a387-8734f87c2080-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:26.190015 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.189963 6273 generic.go:356] "Generic (PLEG): container finished" podID="4fe0da50-df40-4fc7-a387-8734f87c2080" containerID="13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4" exitCode=0 Nov 02 07:02:26.190173 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.190031 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" Nov 02 07:02:26.190173 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.190028 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" event={"ID":"4fe0da50-df40-4fc7-a387-8734f87c2080","Type":"ContainerDied","Data":"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4"} Nov 02 07:02:26.190284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.190193 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6" event={"ID":"4fe0da50-df40-4fc7-a387-8734f87c2080","Type":"ContainerDied","Data":"0c41a7941f0988570983e2326bac04625ed74d46d0b470987e0a3bd975938b4b"} Nov 02 07:02:26.190284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.190220 6273 scope.go:122] "RemoveContainer" containerID="13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4" Nov 02 07:02:26.192698 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.192670 6273 generic.go:356] "Generic (PLEG): container finished" podID="842d153e-144e-4f7f-ad3c-03360e8e2403" containerID="8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b" exitCode=0 Nov 02 07:02:26.192698 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.192754 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" event={"ID":"842d153e-144e-4f7f-ad3c-03360e8e2403","Type":"ContainerDied","Data":"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b"} Nov 02 07:02:26.192956 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.192794 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" event={"ID":"842d153e-144e-4f7f-ad3c-03360e8e2403","Type":"ContainerDied","Data":"490f07e3df9ea0e51f2cffb4403b61bfc495db6a7d2c31a468fd56935cf12767"} Nov 02 07:02:26.192956 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.192870 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z" Nov 02 07:02:26.201517 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.201491 6273 scope.go:122] "RemoveContainer" containerID="13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4" Nov 02 07:02:26.201798 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:26.201779 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4\": container with ID starting with 13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4 not found: ID does not exist" containerID="13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4" Nov 02 07:02:26.201855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.201807 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4"} err="failed to get container status \"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4\": rpc error: code = NotFound desc = could not find container \"13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4\": container with ID starting with 13ed0279bb43160f44bb501c0713cdc894123227bd297fdd1ee0691faa5bebb4 not found: ID does not exist" Nov 02 07:02:26.201855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.201826 6273 scope.go:122] "RemoveContainer" containerID="8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b" Nov 02 07:02:26.210048 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.210026 6273 scope.go:122] "RemoveContainer" containerID="8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b" Nov 02 07:02:26.210347 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:26.210322 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b\": container with ID starting with 8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b not found: ID does not exist" containerID="8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b" Nov 02 07:02:26.210438 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.210353 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b"} err="failed to get container status \"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b\": rpc error: code = NotFound desc = could not find container \"8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b\": container with ID starting with 8edaa9f43a04e088671e2a7615f90ef146f5c65d96fd7b5ddb2971a3a448781b not found: ID does not exist" Nov 02 07:02:26.342313 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.342229 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 07:02:26.343910 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.343882 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6e2657-84c796dc5-rh7x6"] Nov 02 07:02:26.448402 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.448368 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 07:02:26.454294 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:26.454238 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-6e2657-676858d54f-pld7z"] Nov 02 07:02:28.177517 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:28.177484 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe0da50-df40-4fc7-a387-8734f87c2080" path="/var/lib/kubelet/pods/4fe0da50-df40-4fc7-a387-8734f87c2080/volumes" Nov 02 07:02:28.178022 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:28.178006 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842d153e-144e-4f7f-ad3c-03360e8e2403" path="/var/lib/kubelet/pods/842d153e-144e-4f7f-ad3c-03360e8e2403/volumes" Nov 02 07:02:37.177177 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:37.177099 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 07:02:37.223353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:37.223319 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 07:02:38.552484 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:38.552445 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:02:52.197034 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.196970 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" containerID="cri-o://cdaf8dc3805c75a7cbafa6f01633d79b3754ca5b3e4e5a07ca1ba57b457aadd2" gracePeriod=45 Nov 02 07:02:52.242693 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.242652 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" containerID="cri-o://3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0" gracePeriod=45 Nov 02 07:02:52.319660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.319628 6273 generic.go:356] "Generic (PLEG): container finished" podID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerID="cdaf8dc3805c75a7cbafa6f01633d79b3754ca5b3e4e5a07ca1ba57b457aadd2" exitCode=0 Nov 02 07:02:52.319804 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.319702 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" event={"ID":"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb","Type":"ContainerDied","Data":"cdaf8dc3805c75a7cbafa6f01633d79b3754ca5b3e4e5a07ca1ba57b457aadd2"} Nov 02 07:02:52.503366 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.503337 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 07:02:52.522098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.522075 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 07:02:52.594610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594579 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594619 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594641 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594673 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594704 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594730 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.594793 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594774 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594807 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594912 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594910 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.594960 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\" (UniqueName: \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\") pod \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\" (UID: \"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595041 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595115 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\" (UniqueName: \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\") pod \"17ad7deb-e433-4587-a419-cec6305034fc\" (UID: \"17ad7deb-e433-4587-a419-cec6305034fc\") " Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595041 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595280 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595055 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595070 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595240 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595550 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595571 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595586 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595600 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595614 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.596513 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.595629 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.597366 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597334 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.597472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597377 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:52.597472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597411 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.597472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597426 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "kube-api-access-9dt59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:52.597587 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597520 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs" pod "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" (UID: "3b637f7d-0cb9-49ce-8412-8ca46c3be9fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:52.597587 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.597544 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9" pod "17ad7deb-e433-4587-a419-cec6305034fc" (UID: "17ad7deb-e433-4587-a419-cec6305034fc"). InnerVolumeSpecName "kube-api-access-r7cz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:52.697061 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697020 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt59\" (UniqueName: \"kubernetes.io/projected/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-kube-api-access-9dt59\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697061 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697050 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7cz9\" (UniqueName: \"kubernetes.io/projected/17ad7deb-e433-4587-a419-cec6305034fc-kube-api-access-r7cz9\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697061 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697061 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697061 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697070 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17ad7deb-e433-4587-a419-cec6305034fc-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697320 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697078 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17ad7deb-e433-4587-a419-cec6305034fc-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697320 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:52.697087 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:53.325217 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.325179 6273 generic.go:356] "Generic (PLEG): container finished" podID="17ad7deb-e433-4587-a419-cec6305034fc" containerID="3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0" exitCode=0 Nov 02 07:02:53.325652 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.325225 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" Nov 02 07:02:53.325652 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.325255 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" event={"ID":"17ad7deb-e433-4587-a419-cec6305034fc","Type":"ContainerDied","Data":"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0"} Nov 02 07:02:53.325652 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.325299 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx" event={"ID":"17ad7deb-e433-4587-a419-cec6305034fc","Type":"ContainerDied","Data":"9133b0ecf0b982f3dcfd493564cfba4cce8ebd2058afbc59ddf33fb0f5451c91"} Nov 02 07:02:53.325652 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.325316 6273 scope.go:122] "RemoveContainer" containerID="3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0" Nov 02 07:02:53.326827 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.326810 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" Nov 02 07:02:53.326919 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.326819 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6" event={"ID":"3b637f7d-0cb9-49ce-8412-8ca46c3be9fb","Type":"ContainerDied","Data":"90595d3915548564c3a63bc10a02bca1248b6910b1440c3bcf9ece6ad21d9edc"} Nov 02 07:02:53.335691 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.335664 6273 scope.go:122] "RemoveContainer" containerID="3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0" Nov 02 07:02:53.337244 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:02:53.337103 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0\": container with ID starting with 3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0 not found: ID does not exist" containerID="3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0" Nov 02 07:02:53.337244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.337143 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0"} err="failed to get container status \"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0\": rpc error: code = NotFound desc = could not find container \"3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0\": container with ID starting with 3ada9116d9411b44ab6cc2d9c425f7e2ad2e49416fd47b378f0adce4555ce4a0 not found: ID does not exist" Nov 02 07:02:53.337244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.337166 6273 scope.go:122] "RemoveContainer" containerID="cdaf8dc3805c75a7cbafa6f01633d79b3754ca5b3e4e5a07ca1ba57b457aadd2" Nov 02 07:02:53.479435 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.479357 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 07:02:53.525515 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.525485 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-w6-6ff85e-kserve-6bb8dbdb6c-xqmkx"] Nov 02 07:02:53.585533 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.585505 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 07:02:53.588933 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:53.588908 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/distinct-2-w6-6ff85e-kserve-95d6f7f5b-gn4g6"] Nov 02 07:02:54.181345 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:54.181302 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ad7deb-e433-4587-a419-cec6305034fc" path="/var/lib/kubelet/pods/17ad7deb-e433-4587-a419-cec6305034fc/volumes" Nov 02 07:02:54.181923 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:02:54.181900 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" path="/var/lib/kubelet/pods/3b637f7d-0cb9-49ce-8412-8ca46c3be9fb/volumes" Nov 02 07:03:01.087239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.087205 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 07:03:01.087702 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.087408 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" podUID="52310f90-2bf9-43e9-9c48-015c8fa145de" containerName="maas-api" containerID="cri-o://c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3" gracePeriod=30 Nov 02 07:03:01.346011 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.345930 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 07:03:01.368617 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.368581 6273 generic.go:356] "Generic (PLEG): container finished" podID="52310f90-2bf9-43e9-9c48-015c8fa145de" containerID="c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3" exitCode=0 Nov 02 07:03:01.368791 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.368623 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" Nov 02 07:03:01.368791 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.368667 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" event={"ID":"52310f90-2bf9-43e9-9c48-015c8fa145de","Type":"ContainerDied","Data":"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3"} Nov 02 07:03:01.368791 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.368730 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9" event={"ID":"52310f90-2bf9-43e9-9c48-015c8fa145de","Type":"ContainerDied","Data":"1e1e6bff18914c044356d33183bd36e39f09701bba399537813dc5c9b59b859b"} Nov 02 07:03:01.368791 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.368761 6273 scope.go:122] "RemoveContainer" containerID="c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3" Nov 02 07:03:01.377768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.377752 6273 scope.go:122] "RemoveContainer" containerID="c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3" Nov 02 07:03:01.378095 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:03:01.378059 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3\": container with ID starting with c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3 not found: ID does not exist" containerID="c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3" Nov 02 07:03:01.378180 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.378098 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3"} err="failed to get container status \"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3\": rpc error: code = NotFound desc = could not find container \"c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3\": container with ID starting with c11e68c820f49557dadaf1dbe824aa09150daa53a57d4a45ca3e061be5610fe3 not found: ID does not exist" Nov 02 07:03:01.487372 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.487335 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\") pod \"52310f90-2bf9-43e9-9c48-015c8fa145de\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " Nov 02 07:03:01.487571 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.487442 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\" (UniqueName: \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\") pod \"52310f90-2bf9-43e9-9c48-015c8fa145de\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " Nov 02 07:03:01.487571 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.487496 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") pod \"52310f90-2bf9-43e9-9c48-015c8fa145de\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " Nov 02 07:03:01.487571 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.487534 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\") pod \"52310f90-2bf9-43e9-9c48-015c8fa145de\" (UID: \"52310f90-2bf9-43e9-9c48-015c8fa145de\") " Nov 02 07:03:01.487794 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.487769 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca" pod "52310f90-2bf9-43e9-9c48-015c8fa145de" (UID: "52310f90-2bf9-43e9-9c48-015c8fa145de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:03:01.489603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.489569 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls" pod "52310f90-2bf9-43e9-9c48-015c8fa145de" (UID: "52310f90-2bf9-43e9-9c48-015c8fa145de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:01.489603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.489590 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs" pod "52310f90-2bf9-43e9-9c48-015c8fa145de" (UID: "52310f90-2bf9-43e9-9c48-015c8fa145de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:01.489750 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.489673 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454" pod "52310f90-2bf9-43e9-9c48-015c8fa145de" (UID: "52310f90-2bf9-43e9-9c48-015c8fa145de"). InnerVolumeSpecName "kube-api-access-p8454". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:01.588958 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.588913 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8454\" (UniqueName: \"kubernetes.io/projected/52310f90-2bf9-43e9-9c48-015c8fa145de-kube-api-access-p8454\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:01.588958 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.588954 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:01.588958 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.588964 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52310f90-2bf9-43e9-9c48-015c8fa145de-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:01.589236 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.588973 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52310f90-2bf9-43e9-9c48-015c8fa145de-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:01.780874 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.780836 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:03:01.783443 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.783417 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 07:03:01.785199 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:01.785176 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6ff85e-6fc956f5bc-2q6v9"] Nov 02 07:03:02.178384 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:02.178348 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52310f90-2bf9-43e9-9c48-015c8fa145de" path="/var/lib/kubelet/pods/52310f90-2bf9-43e9-9c48-015c8fa145de/volumes" Nov 02 07:03:08.552136 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:03:08.552097 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:03:16.385185 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:16.385148 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:03:38.552484 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:03:38.552446 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:03:40.702708 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.702674 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl"] Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703059 6273 cpu_manager.go:414] "containerMap: removing container" podUID="52310f90-2bf9-43e9-9c48-015c8fa145de" containerName="maas-api" Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703070 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52310f90-2bf9-43e9-9c48-015c8fa145de" containerName="maas-api" Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703080 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703085 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703096 6273 cpu_manager.go:414] "containerMap: removing container" podUID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" containerName="payload-processing" Nov 02 07:03:40.703103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703101 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703111 6273 cpu_manager.go:414] "containerMap: removing container" podUID="842d153e-144e-4f7f-ad3c-03360e8e2403" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703116 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842d153e-144e-4f7f-ad3c-03360e8e2403" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703122 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703128 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703135 6273 cpu_manager.go:414] "containerMap: removing container" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703140 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703148 6273 cpu_manager.go:414] "containerMap: removing container" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703152 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703159 6273 cpu_manager.go:414] "containerMap: removing container" podUID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703163 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703170 6273 cpu_manager.go:414] "containerMap: removing container" podUID="03c60f2f-384b-46b2-8c57-38c03f872876" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703174 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c60f2f-384b-46b2-8c57-38c03f872876" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703184 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703188 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703195 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703199 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703208 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e8a735c0-bc8e-4704-84b6-add16df8fecd" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703213 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a735c0-bc8e-4704-84b6-add16df8fecd" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703225 6273 cpu_manager.go:414] "containerMap: removing container" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703229 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703234 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2eff15b3-d9b1-480c-b43e-795550a9895a" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703238 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eff15b3-d9b1-480c-b43e-795550a9895a" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703244 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8e652264-bcbd-45f3-9a58-0c42115f5529" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703248 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e652264-bcbd-45f3-9a58-0c42115f5529" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703254 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b3c50225-9ac4-4438-a308-6daf361948b5" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703259 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c50225-9ac4-4438-a308-6daf361948b5" containerName="maas-api" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703264 6273 cpu_manager.go:414] "containerMap: removing container" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703268 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703275 6273 cpu_manager.go:414] "containerMap: removing container" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703279 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerName="payload-processing" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703288 6273 cpu_manager.go:414] "containerMap: removing container" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703293 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703298 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" Nov 02 07:03:40.703292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703302 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703311 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703315 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703320 6273 cpu_manager.go:414] "containerMap: removing container" podUID="4fe0da50-df40-4fc7-a387-8734f87c2080" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703324 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe0da50-df40-4fc7-a387-8734f87c2080" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703381 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834064e-dac1-4f44-8fdb-be8b4dfd9224" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703388 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703393 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a735c0-bc8e-4704-84b6-add16df8fecd" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703401 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ad7deb-e433-4587-a419-cec6305034fc" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703406 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eff15b3-d9b1-480c-b43e-795550a9895a" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703412 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e652264-bcbd-45f3-9a58-0c42115f5529" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703417 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c50225-9ac4-4438-a308-6daf361948b5" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703422 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703428 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="736b71f9-b690-4d11-9c9f-5bf91999bf49" containerName="payload-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703435 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1146e3e-9db0-40fb-8b6a-99d8398524e4" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703440 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db6bfd2-6e42-40ed-9fb3-bfa026d677c6" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703444 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b637f7d-0cb9-49ce-8412-8ca46c3be9fb" containerName="main" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703449 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe0da50-df40-4fc7-a387-8734f87c2080" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703454 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="52310f90-2bf9-43e9-9c48-015c8fa145de" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703460 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d438b90-6711-4ce0-bb13-5c3684e8b4b2" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703465 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4b46a8-1510-497f-a0fa-043f3b6b7c67" containerName="payload-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703471 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="842d153e-144e-4f7f-ad3c-03360e8e2403" containerName="payload-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703476 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2dd1a73-8743-4298-b812-eb823cede79d" containerName="payload-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703481 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e60dab-0aaa-4b85-8a50-9b031f9b0fcb" containerName="payload-pre-processing" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703485 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703490 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed43483b-809e-4d2c-8fbc-e89f44d691b0" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703495 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c60f2f-384b-46b2-8c57-38c03f872876" containerName="maas-api" Nov 02 07:03:40.704273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.703501 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bbee33-c01f-4d8e-8664-bb25a4534f44" containerName="maas-api" Nov 02 07:03:40.707788 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.707763 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.760319 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.760289 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl"] Nov 02 07:03:40.805187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.805158 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:03:40.805187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.805182 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:03:40.805353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.805199 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:03:40.805353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.805313 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:03:40.859255 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.859229 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.859349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.859273 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.859349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.859295 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6l5\" (UniqueName: \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.960201 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.960136 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.960201 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.960181 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.960201 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.960201 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6l5\" (UniqueName: \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.960762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.960743 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:40.962652 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:40.962634 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:41.013683 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:41.013658 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6l5\" (UniqueName: \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\") pod \"maas-api-revoke-sub-85237e45-jbcdl\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:41.020623 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:41.020601 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:41.385861 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:41.385780 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl"] Nov 02 07:03:41.390342 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:03:41.390312 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b80116_9cb7_4e0b_8b1e_245a00f9fd46.slice/crio-6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d WatchSource:0}: Error finding container 6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d: Status 404 returned error can't find the container with id 6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d Nov 02 07:03:41.560055 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:41.560020 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" event={"ID":"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46","Type":"ContainerStarted","Data":"6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d"} Nov 02 07:03:42.566599 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:42.566519 6273 generic.go:356] "Generic (PLEG): container finished" podID="c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" containerID="9db322451ec7290fe97b08c15a7bc825e7ad59ea11b59e2539ef334e72cef96c" exitCode=0 Nov 02 07:03:42.566599 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:42.566554 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" event={"ID":"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46","Type":"ContainerDied","Data":"9db322451ec7290fe97b08c15a7bc825e7ad59ea11b59e2539ef334e72cef96c"} Nov 02 07:03:43.707699 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.707671 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:43.784803 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.784775 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\") pod \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " Nov 02 07:03:43.784930 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.784846 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\") pod \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " Nov 02 07:03:43.784930 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.784903 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\" (UniqueName: \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\") pod \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\" (UID: \"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46\") " Nov 02 07:03:43.785211 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.785185 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca" pod "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" (UID: "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:03:43.786914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.786886 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca" pod "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" (UID: "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:43.787034 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.786952 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5" pod "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" (UID: "c5b80116-9cb7-4e0b-8b1e-245a00f9fd46"). InnerVolumeSpecName "kube-api-access-9z6l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:43.886306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.886240 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:43.886306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.886266 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:43.886306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:43.886275 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6l5\" (UniqueName: \"kubernetes.io/projected/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46-kube-api-access-9z6l5\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:44.576747 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:44.576665 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" event={"ID":"c5b80116-9cb7-4e0b-8b1e-245a00f9fd46","Type":"ContainerDied","Data":"6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d"} Nov 02 07:03:44.576747 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:44.576704 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl" Nov 02 07:03:44.576747 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:44.576717 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5953fd4e39b4348ae7e5b2bbdf3bf43febd9a28f37eef2d272598dbf218e3d" Nov 02 07:03:55.574401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574363 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fb8fd585-tz5c5"] Nov 02 07:03:55.576768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574764 6273 cpu_manager.go:414] "containerMap: removing container" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:55.576768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574774 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593d48a2-7e21-441b-b578-02a7f1a6f251" containerName="cleanup" Nov 02 07:03:55.576768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574786 6273 cpu_manager.go:414] "containerMap: removing container" podUID="c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" containerName="revoke-keys" Nov 02 07:03:55.576768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574791 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" containerName="revoke-keys" Nov 02 07:03:55.576768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.574870 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" containerName="revoke-keys" Nov 02 07:03:55.577763 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.577747 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.674463 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.674428 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 07:03:55.674463 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.674438 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 07:03:55.674463 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.674453 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 07:03:55.674713 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.674657 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prjsd\"" Nov 02 07:03:55.675980 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.675962 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-tz5c5"] Nov 02 07:03:55.692109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.692083 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-webhook-cert\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.692252 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.692125 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-metrics-certs\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.692300 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.692257 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jwp\" (UniqueName: \"kubernetes.io/projected/a7695fda-4da8-4451-956c-aa9f9a780c69-kube-api-access-c6jwp\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.793768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.793732 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-webhook-cert\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.793919 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.793803 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-metrics-certs\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.793919 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.793905 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jwp\" (UniqueName: \"kubernetes.io/projected/a7695fda-4da8-4451-956c-aa9f9a780c69-kube-api-access-c6jwp\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.796112 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.796086 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-webhook-cert\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.796233 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.796215 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7695fda-4da8-4451-956c-aa9f9a780c69-metrics-certs\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.848128 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.848074 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jwp\" (UniqueName: \"kubernetes.io/projected/a7695fda-4da8-4451-956c-aa9f9a780c69-kube-api-access-c6jwp\") pod \"maas-controller-59fb8fd585-tz5c5\" (UID: \"a7695fda-4da8-4451-956c-aa9f9a780c69\") " pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:55.891948 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:55.891925 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:56.257195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:56.256959 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-tz5c5"] Nov 02 07:03:56.259452 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:03:56.259423 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7695fda_4da8_4451_956c_aa9f9a780c69.slice/crio-fe8af9c55b7dae520f98a599149a49c738ee2d7865150fca9292a1c075cb5fa6 WatchSource:0}: Error finding container fe8af9c55b7dae520f98a599149a49c738ee2d7865150fca9292a1c075cb5fa6: Status 404 returned error can't find the container with id fe8af9c55b7dae520f98a599149a49c738ee2d7865150fca9292a1c075cb5fa6 Nov 02 07:03:56.260836 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:56.260807 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:03:56.633797 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:56.633723 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" event={"ID":"a7695fda-4da8-4451-956c-aa9f9a780c69","Type":"ContainerStarted","Data":"fe8af9c55b7dae520f98a599149a49c738ee2d7865150fca9292a1c075cb5fa6"} Nov 02 07:03:58.645732 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:58.645690 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" event={"ID":"a7695fda-4da8-4451-956c-aa9f9a780c69","Type":"ContainerStarted","Data":"11eb123306e1a32f60bbc9c4632b3852113237b9c77f2bd3f0c6992b3d1608a0"} Nov 02 07:03:58.646207 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:58.645768 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:03:58.774633 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:03:58.774539 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" podStartSLOduration=1.47652734 podStartE2EDuration="3.774525721s" totalImagesPullingTime="2.297998381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:58.774126288 +0000 UTC m=+2792.747460964" watchObservedRunningTime="2026-11-02 07:03:58.774525721 +0000 UTC m=+2792.747860404" Nov 02 07:04:08.559902 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:04:08.559814 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:04:09.653492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:09.653462 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fb8fd585-tz5c5" Nov 02 07:04:38.552470 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:04:38.552436 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:04:39.190328 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:39.190295 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:04:47.771762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.771727 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9"] Nov 02 07:04:47.777619 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.777601 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.797133 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.797106 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvbc\" (UniqueName: \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.797265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.797142 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.797265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.797177 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.830178 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.830152 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9"] Nov 02 07:04:47.874678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.874647 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:04:47.874830 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.874697 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:04:47.874830 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.874697 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:04:47.874830 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.874772 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:04:47.897914 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.897883 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.898086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.898014 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvbc\" (UniqueName: \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.898086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.898047 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.898674 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.898651 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.900249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.900219 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:47.951762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:47.951733 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvbc\" (UniqueName: \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\") pod \"maas-api-revoke-sub-c1e271a9-6vbc9\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:48.091450 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:48.091366 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:48.458363 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:48.458328 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9"] Nov 02 07:04:48.462290 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:04:48.462261 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10c58bf_ba92_4fab_8e05_7db00d148982.slice/crio-947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792 WatchSource:0}: Error finding container 947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792: Status 404 returned error can't find the container with id 947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792 Nov 02 07:04:48.914100 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:48.914063 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" event={"ID":"f10c58bf-ba92-4fab-8e05-7db00d148982","Type":"ContainerStarted","Data":"9e7b4438c4223ae1b6676a396325e83524a109a0bf1b7510a129da7e087b0115"} Nov 02 07:04:48.914455 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:48.914124 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" event={"ID":"f10c58bf-ba92-4fab-8e05-7db00d148982","Type":"ContainerStarted","Data":"947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792"} Nov 02 07:04:49.021780 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:49.021677 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" podStartSLOduration=2.021663839 podStartE2EDuration="2.021663839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:04:49.020586467 +0000 UTC m=+2842.993921171" watchObservedRunningTime="2026-11-02 07:04:49.021663839 +0000 UTC m=+2842.994998501" Nov 02 07:04:49.920021 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:49.919969 6273 generic.go:356] "Generic (PLEG): container finished" podID="f10c58bf-ba92-4fab-8e05-7db00d148982" containerID="9e7b4438c4223ae1b6676a396325e83524a109a0bf1b7510a129da7e087b0115" exitCode=0 Nov 02 07:04:49.920418 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:49.920054 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" event={"ID":"f10c58bf-ba92-4fab-8e05-7db00d148982","Type":"ContainerDied","Data":"9e7b4438c4223ae1b6676a396325e83524a109a0bf1b7510a129da7e087b0115"} Nov 02 07:04:51.058335 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.058311 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:51.130854 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.130828 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\") pod \"f10c58bf-ba92-4fab-8e05-7db00d148982\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " Nov 02 07:04:51.131030 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.130867 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\") pod \"f10c58bf-ba92-4fab-8e05-7db00d148982\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " Nov 02 07:04:51.131030 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.130962 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\" (UniqueName: \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\") pod \"f10c58bf-ba92-4fab-8e05-7db00d148982\" (UID: \"f10c58bf-ba92-4fab-8e05-7db00d148982\") " Nov 02 07:04:51.131277 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.131255 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca" pod "f10c58bf-ba92-4fab-8e05-7db00d148982" (UID: "f10c58bf-ba92-4fab-8e05-7db00d148982"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:04:51.133092 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.133068 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca" pod "f10c58bf-ba92-4fab-8e05-7db00d148982" (UID: "f10c58bf-ba92-4fab-8e05-7db00d148982"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:04:51.133195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.133092 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc" pod "f10c58bf-ba92-4fab-8e05-7db00d148982" (UID: "f10c58bf-ba92-4fab-8e05-7db00d148982"). InnerVolumeSpecName "kube-api-access-9rvbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:04:51.232151 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.232073 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:04:51.232151 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.232102 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10c58bf-ba92-4fab-8e05-7db00d148982-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:04:51.232151 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.232114 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvbc\" (UniqueName: \"kubernetes.io/projected/f10c58bf-ba92-4fab-8e05-7db00d148982-kube-api-access-9rvbc\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:04:51.934196 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.934164 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" Nov 02 07:04:51.934401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.934205 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9" event={"ID":"f10c58bf-ba92-4fab-8e05-7db00d148982","Type":"ContainerDied","Data":"947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792"} Nov 02 07:04:51.934401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:04:51.934250 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947b6dae96c844b54550b1e1296b07e73df5c719b778b04f11c71b571a43a792" Nov 02 07:05:00.579695 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:00.579659 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:05:08.552851 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:05:08.552806 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:05:10.782685 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:10.782645 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:05:18.758432 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.758399 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9"] Nov 02 07:05:18.758860 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.758846 6273 cpu_manager.go:414] "containerMap: removing container" podUID="f10c58bf-ba92-4fab-8e05-7db00d148982" containerName="revoke-keys" Nov 02 07:05:18.758920 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.758861 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10c58bf-ba92-4fab-8e05-7db00d148982" containerName="revoke-keys" Nov 02 07:05:18.758977 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.758963 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10c58bf-ba92-4fab-8e05-7db00d148982" containerName="revoke-keys" Nov 02 07:05:18.762505 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.762479 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.819778 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.819723 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2lg\" (UniqueName: \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.819778 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.819782 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.820079 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.819934 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.858296 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.858258 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:05:18.858454 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.858297 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:05:18.858454 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.858303 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:05:18.858454 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.858354 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:05:18.859752 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.859730 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9"] Nov 02 07:05:18.921578 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.921552 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2lg\" (UniqueName: \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.921739 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.921589 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.921871 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.921852 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.922264 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.922242 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.924259 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.924237 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:18.975454 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:18.975420 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2lg\" (UniqueName: \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\") pod \"maas-api-revoke-sub-2e1f5b5f-vgbm9\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:19.076455 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:19.076377 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:19.441072 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:19.441040 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9"] Nov 02 07:05:19.444829 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:05:19.444792 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263f3a9a_9ceb_4e4b_9981_c0c706784795.slice/crio-1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f WatchSource:0}: Error finding container 1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f: Status 404 returned error can't find the container with id 1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f Nov 02 07:05:20.083657 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:20.083569 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" event={"ID":"263f3a9a-9ceb-4e4b-9981-c0c706784795","Type":"ContainerStarted","Data":"8074d6980d27a5a17e97930516ca4719c34a6ffdfe279839c5fd41914b14cb07"} Nov 02 07:05:20.083657 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:20.083620 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" event={"ID":"263f3a9a-9ceb-4e4b-9981-c0c706784795","Type":"ContainerStarted","Data":"1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f"} Nov 02 07:05:20.191121 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:20.191071 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" podStartSLOduration=2.191054699 podStartE2EDuration="2.191054699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:05:20.191004441 +0000 UTC m=+2874.164339126" watchObservedRunningTime="2026-11-02 07:05:20.191054699 +0000 UTC m=+2874.164389384" Nov 02 07:05:21.089402 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:21.089365 6273 generic.go:356] "Generic (PLEG): container finished" podID="263f3a9a-9ceb-4e4b-9981-c0c706784795" containerID="8074d6980d27a5a17e97930516ca4719c34a6ffdfe279839c5fd41914b14cb07" exitCode=0 Nov 02 07:05:21.089803 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:21.089454 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" event={"ID":"263f3a9a-9ceb-4e4b-9981-c0c706784795","Type":"ContainerDied","Data":"8074d6980d27a5a17e97930516ca4719c34a6ffdfe279839c5fd41914b14cb07"} Nov 02 07:05:22.232968 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.232942 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:22.257203 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.257108 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\") pod \"263f3a9a-9ceb-4e4b-9981-c0c706784795\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " Nov 02 07:05:22.257333 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.257288 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\" (UniqueName: \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\") pod \"263f3a9a-9ceb-4e4b-9981-c0c706784795\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " Nov 02 07:05:22.257433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.257407 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca" pod "263f3a9a-9ceb-4e4b-9981-c0c706784795" (UID: "263f3a9a-9ceb-4e4b-9981-c0c706784795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:05:22.257433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.257424 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\") pod \"263f3a9a-9ceb-4e4b-9981-c0c706784795\" (UID: \"263f3a9a-9ceb-4e4b-9981-c0c706784795\") " Nov 02 07:05:22.257781 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.257760 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:22.259441 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.259418 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca" pod "263f3a9a-9ceb-4e4b-9981-c0c706784795" (UID: "263f3a9a-9ceb-4e4b-9981-c0c706784795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:05:22.259529 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.259469 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg" pod "263f3a9a-9ceb-4e4b-9981-c0c706784795" (UID: "263f3a9a-9ceb-4e4b-9981-c0c706784795"). InnerVolumeSpecName "kube-api-access-dt2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:05:22.358457 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.358391 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2lg\" (UniqueName: \"kubernetes.io/projected/263f3a9a-9ceb-4e4b-9981-c0c706784795-kube-api-access-dt2lg\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:22.358457 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:22.358423 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263f3a9a-9ceb-4e4b-9981-c0c706784795-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:23.101380 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:23.101347 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" Nov 02 07:05:23.101380 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:23.101358 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9" event={"ID":"263f3a9a-9ceb-4e4b-9981-c0c706784795","Type":"ContainerDied","Data":"1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f"} Nov 02 07:05:23.101593 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:23.101453 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1baf634cfafc3f1d2a33df1ef6bf8d337038196c673ac06d97bb468c3cb0483f" Nov 02 07:05:31.019931 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.019894 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c"] Nov 02 07:05:31.020423 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.020406 6273 cpu_manager.go:414] "containerMap: removing container" podUID="263f3a9a-9ceb-4e4b-9981-c0c706784795" containerName="revoke-keys" Nov 02 07:05:31.020500 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.020424 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263f3a9a-9ceb-4e4b-9981-c0c706784795" containerName="revoke-keys" Nov 02 07:05:31.020558 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.020524 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="263f3a9a-9ceb-4e4b-9981-c0c706784795" containerName="revoke-keys" Nov 02 07:05:31.028306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.028285 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.078460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.078429 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c"] Nov 02 07:05:31.122560 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.122508 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:05:31.122701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.122567 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:05:31.123682 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.123657 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:05:31.123807 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.123712 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:05:31.141700 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.141677 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.141848 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.141713 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn8v\" (UniqueName: \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.141920 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.141888 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.242935 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.242911 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.243135 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.242949 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn8v\" (UniqueName: \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.243135 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.243049 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.243759 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.243736 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.245361 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.245342 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.296385 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.296330 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn8v\" (UniqueName: \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\") pod \"maas-api-revoke-sub-ec843018-4h22c\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.341333 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.341307 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:31.705504 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:31.705473 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c"] Nov 02 07:05:31.709379 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:05:31.709353 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1525379_ad0c_4e8d_ac63_33f2ba823694.slice/crio-ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957 WatchSource:0}: Error finding container ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957: Status 404 returned error can't find the container with id ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957 Nov 02 07:05:32.146873 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:32.146836 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" event={"ID":"d1525379-ad0c-4e8d-ac63-33f2ba823694","Type":"ContainerStarted","Data":"36c3ca97dc9fcc1b540ab8450f16f7e10317aea8223cfd912f9a0b953bbb4ce4"} Nov 02 07:05:32.147409 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:32.146889 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" event={"ID":"d1525379-ad0c-4e8d-ac63-33f2ba823694","Type":"ContainerStarted","Data":"ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957"} Nov 02 07:05:32.254685 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:32.254586 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" podStartSLOduration=2.254571264 podStartE2EDuration="2.254571264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:05:32.253525261 +0000 UTC m=+2886.226859946" watchObservedRunningTime="2026-11-02 07:05:32.254571264 +0000 UTC m=+2886.227905945" Nov 02 07:05:33.154622 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:33.154592 6273 generic.go:356] "Generic (PLEG): container finished" podID="d1525379-ad0c-4e8d-ac63-33f2ba823694" containerID="36c3ca97dc9fcc1b540ab8450f16f7e10317aea8223cfd912f9a0b953bbb4ce4" exitCode=0 Nov 02 07:05:33.155012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:33.154633 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" event={"ID":"d1525379-ad0c-4e8d-ac63-33f2ba823694","Type":"ContainerDied","Data":"36c3ca97dc9fcc1b540ab8450f16f7e10317aea8223cfd912f9a0b953bbb4ce4"} Nov 02 07:05:34.296795 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.296773 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:34.373048 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.373021 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\" (UniqueName: \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\") pod \"d1525379-ad0c-4e8d-ac63-33f2ba823694\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " Nov 02 07:05:34.373195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.373080 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\") pod \"d1525379-ad0c-4e8d-ac63-33f2ba823694\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " Nov 02 07:05:34.373195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.373170 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\") pod \"d1525379-ad0c-4e8d-ac63-33f2ba823694\" (UID: \"d1525379-ad0c-4e8d-ac63-33f2ba823694\") " Nov 02 07:05:34.373528 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.373503 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca" pod "d1525379-ad0c-4e8d-ac63-33f2ba823694" (UID: "d1525379-ad0c-4e8d-ac63-33f2ba823694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:05:34.375036 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.375013 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v" pod "d1525379-ad0c-4e8d-ac63-33f2ba823694" (UID: "d1525379-ad0c-4e8d-ac63-33f2ba823694"). InnerVolumeSpecName "kube-api-access-kmn8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:05:34.375202 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.375186 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca" pod "d1525379-ad0c-4e8d-ac63-33f2ba823694" (UID: "d1525379-ad0c-4e8d-ac63-33f2ba823694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:05:34.474349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.474269 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:34.474349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.474294 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmn8v\" (UniqueName: \"kubernetes.io/projected/d1525379-ad0c-4e8d-ac63-33f2ba823694-kube-api-access-kmn8v\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:34.474349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.474305 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1525379-ad0c-4e8d-ac63-33f2ba823694-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:05:34.866118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:34.866043 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:05:35.167033 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:35.166974 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" event={"ID":"d1525379-ad0c-4e8d-ac63-33f2ba823694","Type":"ContainerDied","Data":"ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957"} Nov 02 07:05:35.167233 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:35.167061 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab81565aa33de56ffdf5fcb89cbba00d40f65c7eb25c355e4079eb76451f7957" Nov 02 07:05:35.167233 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:35.167014 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c" Nov 02 07:05:38.551851 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:05:38.551812 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:05:48.072722 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:05:48.072688 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:06:06.230946 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.230915 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw"] Nov 02 07:06:06.231344 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.231333 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d1525379-ad0c-4e8d-ac63-33f2ba823694" containerName="revoke-keys" Nov 02 07:06:06.231395 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.231345 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1525379-ad0c-4e8d-ac63-33f2ba823694" containerName="revoke-keys" Nov 02 07:06:06.231450 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.231439 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1525379-ad0c-4e8d-ac63-33f2ba823694" containerName="revoke-keys" Nov 02 07:06:06.234448 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.234432 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.269156 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.269131 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.269291 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.269271 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.269433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.269421 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpf2\" (UniqueName: \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.288564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.288540 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw"] Nov 02 07:06:06.332933 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.332908 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:06:06.333074 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.332956 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:06:06.333074 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.332977 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:06:06.333074 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.333020 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:06:06.370647 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.370623 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.370772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.370668 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpf2\" (UniqueName: \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.370772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.370706 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.371277 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.371259 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.373044 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.373027 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.424749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.424726 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpf2\" (UniqueName: \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\") pod \"maas-api-revoke-sub-046e1343-5xkjw\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.547404 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.547345 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:06.916104 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:06.916075 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw"] Nov 02 07:06:06.919341 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:06:06.919315 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda725dedc_adfb_4530_8592_71b9b758a7f3.slice/crio-357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e WatchSource:0}: Error finding container 357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e: Status 404 returned error can't find the container with id 357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e Nov 02 07:06:07.334482 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:07.334448 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" event={"ID":"a725dedc-adfb-4530-8592-71b9b758a7f3","Type":"ContainerStarted","Data":"ace0f9c2e98a83ced0506e39eac01688ce3a91367bde6cb743d8d7e307f1d67a"} Nov 02 07:06:07.334883 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:07.334503 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" event={"ID":"a725dedc-adfb-4530-8592-71b9b758a7f3","Type":"ContainerStarted","Data":"357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e"} Nov 02 07:06:07.443451 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:07.443399 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" podStartSLOduration=1.443384879 podStartE2EDuration="1.443384879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:06:07.441633907 +0000 UTC m=+2921.414968590" watchObservedRunningTime="2026-11-02 07:06:07.443384879 +0000 UTC m=+2921.416719628" Nov 02 07:06:08.340512 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:08.340472 6273 generic.go:356] "Generic (PLEG): container finished" podID="a725dedc-adfb-4530-8592-71b9b758a7f3" containerID="ace0f9c2e98a83ced0506e39eac01688ce3a91367bde6cb743d8d7e307f1d67a" exitCode=0 Nov 02 07:06:08.341108 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:08.340549 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" event={"ID":"a725dedc-adfb-4530-8592-71b9b758a7f3","Type":"ContainerDied","Data":"ace0f9c2e98a83ced0506e39eac01688ce3a91367bde6cb743d8d7e307f1d67a"} Nov 02 07:06:08.551697 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:06:08.551664 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:06:09.479326 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.479302 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:09.604337 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.604257 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\") pod \"a725dedc-adfb-4530-8592-71b9b758a7f3\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " Nov 02 07:06:09.604337 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.604299 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\") pod \"a725dedc-adfb-4530-8592-71b9b758a7f3\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " Nov 02 07:06:09.604571 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.604419 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\" (UniqueName: \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\") pod \"a725dedc-adfb-4530-8592-71b9b758a7f3\" (UID: \"a725dedc-adfb-4530-8592-71b9b758a7f3\") " Nov 02 07:06:09.604678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.604654 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca" pod "a725dedc-adfb-4530-8592-71b9b758a7f3" (UID: "a725dedc-adfb-4530-8592-71b9b758a7f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:06:09.604769 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.604751 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:09.606500 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.606476 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca" pod "a725dedc-adfb-4530-8592-71b9b758a7f3" (UID: "a725dedc-adfb-4530-8592-71b9b758a7f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:06:09.606603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.606517 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2" pod "a725dedc-adfb-4530-8592-71b9b758a7f3" (UID: "a725dedc-adfb-4530-8592-71b9b758a7f3"). InnerVolumeSpecName "kube-api-access-wqpf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:06:09.705531 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.705500 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a725dedc-adfb-4530-8592-71b9b758a7f3-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:09.705531 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:09.705525 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqpf2\" (UniqueName: \"kubernetes.io/projected/a725dedc-adfb-4530-8592-71b9b758a7f3-kube-api-access-wqpf2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:10.351536 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:10.351508 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" Nov 02 07:06:10.351702 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:10.351507 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw" event={"ID":"a725dedc-adfb-4530-8592-71b9b758a7f3","Type":"ContainerDied","Data":"357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e"} Nov 02 07:06:10.351702 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:10.351654 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357303af6bbed1a941e8729edba53e588e63d8182d9e25338de44a0b3998f79e" Nov 02 07:06:16.941035 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:16.941000 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz"] Nov 02 07:06:16.941471 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:16.941433 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a725dedc-adfb-4530-8592-71b9b758a7f3" containerName="revoke-keys" Nov 02 07:06:16.941471 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:16.941445 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a725dedc-adfb-4530-8592-71b9b758a7f3" containerName="revoke-keys" Nov 02 07:06:16.941551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:16.941503 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a725dedc-adfb-4530-8592-71b9b758a7f3" containerName="revoke-keys" Nov 02 07:06:16.944727 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:16.944707 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.042869 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.042836 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:06:17.043109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.043020 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:06:17.043109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.043030 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:06:17.043265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.043209 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:06:17.044827 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.044798 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz"] Nov 02 07:06:17.080183 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.080162 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.080300 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.080198 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76xl\" (UniqueName: \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.080347 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.080320 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.181903 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.181867 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.182154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.181937 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.182154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.181964 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76xl\" (UniqueName: \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.182646 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.182618 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.184300 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.184284 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.236053 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.235966 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76xl\" (UniqueName: \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\") pod \"maas-api-revoke-sub-d3fba056-v8zqz\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.257614 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.257588 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:17.634264 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:17.634239 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz"] Nov 02 07:06:17.637209 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:06:17.637180 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23efa26a_dcaa_423b_a7e4_6ff90f05c06b.slice/crio-ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a WatchSource:0}: Error finding container ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a: Status 404 returned error can't find the container with id ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a Nov 02 07:06:18.392201 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:18.392112 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" event={"ID":"23efa26a-dcaa-423b-a7e4-6ff90f05c06b","Type":"ContainerStarted","Data":"eae9439769a908c24090001693eb523362004e6f839c8cd6b739f313012041b3"} Nov 02 07:06:18.392201 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:18.392167 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" event={"ID":"23efa26a-dcaa-423b-a7e4-6ff90f05c06b","Type":"ContainerStarted","Data":"ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a"} Nov 02 07:06:18.502187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:18.502139 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" podStartSLOduration=2.502126307 podStartE2EDuration="2.502126307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:06:18.501430704 +0000 UTC m=+2932.474765391" watchObservedRunningTime="2026-11-02 07:06:18.502126307 +0000 UTC m=+2932.475460990" Nov 02 07:06:19.084116 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:19.084072 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:06:19.398659 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:19.398571 6273 generic.go:356] "Generic (PLEG): container finished" podID="23efa26a-dcaa-423b-a7e4-6ff90f05c06b" containerID="eae9439769a908c24090001693eb523362004e6f839c8cd6b739f313012041b3" exitCode=0 Nov 02 07:06:19.399080 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:19.398653 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" event={"ID":"23efa26a-dcaa-423b-a7e4-6ff90f05c06b","Type":"ContainerDied","Data":"eae9439769a908c24090001693eb523362004e6f839c8cd6b739f313012041b3"} Nov 02 07:06:20.775089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.772171 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:20.939650 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.939619 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\") pod \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " Nov 02 07:06:20.939818 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.939656 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\" (UniqueName: \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\") pod \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " Nov 02 07:06:20.939818 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.939679 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\") pod \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\" (UID: \"23efa26a-dcaa-423b-a7e4-6ff90f05c06b\") " Nov 02 07:06:20.940157 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.940123 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca" pod "23efa26a-dcaa-423b-a7e4-6ff90f05c06b" (UID: "23efa26a-dcaa-423b-a7e4-6ff90f05c06b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:06:20.941926 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.941895 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl" pod "23efa26a-dcaa-423b-a7e4-6ff90f05c06b" (UID: "23efa26a-dcaa-423b-a7e4-6ff90f05c06b"). InnerVolumeSpecName "kube-api-access-x76xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:06:20.942053 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:20.942003 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca" pod "23efa26a-dcaa-423b-a7e4-6ff90f05c06b" (UID: "23efa26a-dcaa-423b-a7e4-6ff90f05c06b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:06:21.041183 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.041086 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:21.041183 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.041132 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x76xl\" (UniqueName: \"kubernetes.io/projected/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-kube-api-access-x76xl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:21.041183 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.041145 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23efa26a-dcaa-423b-a7e4-6ff90f05c06b-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:21.422476 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.422446 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" Nov 02 07:06:21.422660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.422449 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz" event={"ID":"23efa26a-dcaa-423b-a7e4-6ff90f05c06b","Type":"ContainerDied","Data":"ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a"} Nov 02 07:06:21.422660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:21.422589 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab1a4053123d52939534e6972202f9efbe65fd5066cf626fbe805714d0a23e4a" Nov 02 07:06:29.675736 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:29.675702 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:06:38.585453 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:06:38.585396 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:06:41.646243 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:41.646211 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:06:51.723336 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.723303 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw"] Nov 02 07:06:51.723828 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.723711 6273 cpu_manager.go:414] "containerMap: removing container" podUID="23efa26a-dcaa-423b-a7e4-6ff90f05c06b" containerName="revoke-keys" Nov 02 07:06:51.723828 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.723722 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23efa26a-dcaa-423b-a7e4-6ff90f05c06b" containerName="revoke-keys" Nov 02 07:06:51.723828 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.723803 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="23efa26a-dcaa-423b-a7e4-6ff90f05c06b" containerName="revoke-keys" Nov 02 07:06:51.727154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.727138 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.824089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.824061 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:06:51.824241 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.824097 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:06:51.824241 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.824148 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:06:51.824241 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.824201 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:06:51.825806 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.825785 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw"] Nov 02 07:06:51.856491 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.856470 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jl2\" (UniqueName: \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.856628 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.856548 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.856628 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.856615 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.957760 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.957732 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jl2\" (UniqueName: \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.957898 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.957842 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.957898 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.957873 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.958458 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.958433 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:51.960246 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:51.960229 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:52.012281 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:52.012219 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jl2\" (UniqueName: \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\") pod \"maas-api-revoke-sub-ae594979-k4pqw\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:52.039752 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:52.039728 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:52.402430 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:52.402404 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw"] Nov 02 07:06:52.405499 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:06:52.405472 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cdb369_b36e_494d_8406_5002f1b5cfe0.slice/crio-d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603 WatchSource:0}: Error finding container d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603: Status 404 returned error can't find the container with id d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603 Nov 02 07:06:52.596478 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:52.596435 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" event={"ID":"32cdb369-b36e-494d-8406-5002f1b5cfe0","Type":"ContainerStarted","Data":"d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603"} Nov 02 07:06:53.602583 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:53.602555 6273 generic.go:356] "Generic (PLEG): container finished" podID="32cdb369-b36e-494d-8406-5002f1b5cfe0" containerID="061ac7c338376c650f1966c8bcb09aa38804a4dde8cb5c8d6c1f370127893988" exitCode=0 Nov 02 07:06:53.602969 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:53.602612 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" event={"ID":"32cdb369-b36e-494d-8406-5002f1b5cfe0","Type":"ContainerDied","Data":"061ac7c338376c650f1966c8bcb09aa38804a4dde8cb5c8d6c1f370127893988"} Nov 02 07:06:54.751358 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.751331 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:54.886872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.886791 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\") pod \"32cdb369-b36e-494d-8406-5002f1b5cfe0\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " Nov 02 07:06:54.887062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.886897 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\" (UniqueName: \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\") pod \"32cdb369-b36e-494d-8406-5002f1b5cfe0\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " Nov 02 07:06:54.887062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.887024 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\") pod \"32cdb369-b36e-494d-8406-5002f1b5cfe0\" (UID: \"32cdb369-b36e-494d-8406-5002f1b5cfe0\") " Nov 02 07:06:54.887343 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.887311 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca" pod "32cdb369-b36e-494d-8406-5002f1b5cfe0" (UID: "32cdb369-b36e-494d-8406-5002f1b5cfe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:06:54.889031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.888980 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2" pod "32cdb369-b36e-494d-8406-5002f1b5cfe0" (UID: "32cdb369-b36e-494d-8406-5002f1b5cfe0"). InnerVolumeSpecName "kube-api-access-f5jl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:06:54.889031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.888982 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca" pod "32cdb369-b36e-494d-8406-5002f1b5cfe0" (UID: "32cdb369-b36e-494d-8406-5002f1b5cfe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:06:54.988836 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.988793 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:54.988836 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.988825 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdb369-b36e-494d-8406-5002f1b5cfe0-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:54.988836 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:54.988844 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jl2\" (UniqueName: \"kubernetes.io/projected/32cdb369-b36e-494d-8406-5002f1b5cfe0-kube-api-access-f5jl2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:06:55.621523 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:55.621491 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" Nov 02 07:06:55.621523 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:55.621503 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw" event={"ID":"32cdb369-b36e-494d-8406-5002f1b5cfe0","Type":"ContainerDied","Data":"d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603"} Nov 02 07:06:55.621732 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:06:55.621552 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3147f46668e80316c78539b05bc618bd40f32031af29ad322c71c189e2a5603" Nov 02 07:07:04.591363 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:04.591280 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:07:08.554253 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:07:08.554218 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:07:13.843251 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:13.843215 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:07:19.768567 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.768537 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n"] Nov 02 07:07:19.768996 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.768971 6273 cpu_manager.go:414] "containerMap: removing container" podUID="32cdb369-b36e-494d-8406-5002f1b5cfe0" containerName="revoke-keys" Nov 02 07:07:19.769048 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.768998 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cdb369-b36e-494d-8406-5002f1b5cfe0" containerName="revoke-keys" Nov 02 07:07:19.769089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.769073 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cdb369-b36e-494d-8406-5002f1b5cfe0" containerName="revoke-keys" Nov 02 07:07:19.774189 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.774163 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.827824 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.827688 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n"] Nov 02 07:07:19.855623 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.855595 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.855778 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.855755 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.855910 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.855890 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsnd\" (UniqueName: \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.871838 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.871813 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:07:19.871838 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.871814 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:07:19.872012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.871838 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:07:19.872012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.871866 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:07:19.957264 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.957234 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsnd\" (UniqueName: \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.957408 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.957280 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.957408 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.957381 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.958071 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.958054 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:19.959670 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:19.959647 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:20.011488 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.011468 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsnd\" (UniqueName: \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\") pod \"maas-api-revoke-sub-a4aef7d7-hkl4n\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:20.087029 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.086951 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:20.458103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.458069 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n"] Nov 02 07:07:20.465363 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:07:20.465337 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5ebea7_99cf_4267_ad9e_048bf79dd6df.slice/crio-8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8 WatchSource:0}: Error finding container 8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8: Status 404 returned error can't find the container with id 8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8 Nov 02 07:07:20.768810 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.768781 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" event={"ID":"5e5ebea7-99cf-4267-ad9e-048bf79dd6df","Type":"ContainerStarted","Data":"60f862aad4ea54f53d00ce8561fe89be076cb7e354694347c64d670773540814"} Nov 02 07:07:20.769241 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.768830 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" event={"ID":"5e5ebea7-99cf-4267-ad9e-048bf79dd6df","Type":"ContainerStarted","Data":"8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8"} Nov 02 07:07:20.877635 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:20.877583 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" podStartSLOduration=1.877566645 podStartE2EDuration="1.877566645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:07:20.875712493 +0000 UTC m=+2994.849047191" watchObservedRunningTime="2026-11-02 07:07:20.877566645 +0000 UTC m=+2994.850901328" Nov 02 07:07:21.783602 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:21.783567 6273 generic.go:356] "Generic (PLEG): container finished" podID="5e5ebea7-99cf-4267-ad9e-048bf79dd6df" containerID="60f862aad4ea54f53d00ce8561fe89be076cb7e354694347c64d670773540814" exitCode=0 Nov 02 07:07:21.784031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:21.783649 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" event={"ID":"5e5ebea7-99cf-4267-ad9e-048bf79dd6df","Type":"ContainerDied","Data":"60f862aad4ea54f53d00ce8561fe89be076cb7e354694347c64d670773540814"} Nov 02 07:07:22.920444 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:22.920417 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:23.092062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.091947 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\") pod \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " Nov 02 07:07:23.092062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.092017 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\" (UniqueName: \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\") pod \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " Nov 02 07:07:23.092305 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.092084 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\") pod \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\" (UID: \"5e5ebea7-99cf-4267-ad9e-048bf79dd6df\") " Nov 02 07:07:23.092375 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.092302 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca" pod "5e5ebea7-99cf-4267-ad9e-048bf79dd6df" (UID: "5e5ebea7-99cf-4267-ad9e-048bf79dd6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:07:23.094204 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.094170 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd" pod "5e5ebea7-99cf-4267-ad9e-048bf79dd6df" (UID: "5e5ebea7-99cf-4267-ad9e-048bf79dd6df"). InnerVolumeSpecName "kube-api-access-wvsnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:07:23.094339 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.094247 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca" pod "5e5ebea7-99cf-4267-ad9e-048bf79dd6df" (UID: "5e5ebea7-99cf-4267-ad9e-048bf79dd6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:07:23.192889 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.192857 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:23.192889 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.192883 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:23.193086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.192899 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsnd\" (UniqueName: \"kubernetes.io/projected/5e5ebea7-99cf-4267-ad9e-048bf79dd6df-kube-api-access-wvsnd\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:23.795020 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.794969 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" event={"ID":"5e5ebea7-99cf-4267-ad9e-048bf79dd6df","Type":"ContainerDied","Data":"8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8"} Nov 02 07:07:23.795234 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.795043 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b32d756257e1a7b0427a876edb99d4c6ba05af1b91db8c6cd70112fba5f5ce8" Nov 02 07:07:23.795234 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:23.795051 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n" Nov 02 07:07:32.470154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:32.470122 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:07:38.552905 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:07:38.552868 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:07:44.289894 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:44.289858 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:07:50.465444 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.465407 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq"] Nov 02 07:07:50.465847 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.465833 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5e5ebea7-99cf-4267-ad9e-048bf79dd6df" containerName="revoke-keys" Nov 02 07:07:50.465891 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.465847 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5ebea7-99cf-4267-ad9e-048bf79dd6df" containerName="revoke-keys" Nov 02 07:07:50.465937 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.465928 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5ebea7-99cf-4267-ad9e-048bf79dd6df" containerName="revoke-keys" Nov 02 07:07:50.470185 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.470165 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.496293 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.496269 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mrs\" (UniqueName: \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.496415 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.496309 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.496415 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.496375 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.523783 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.523755 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq"] Nov 02 07:07:50.567809 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.567782 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:07:50.567809 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.567781 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:07:50.568013 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.567824 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:07:50.568013 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.567896 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:07:50.597266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.597237 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mrs\" (UniqueName: \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.597384 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.597282 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.597384 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.597318 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.597940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.597922 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.599602 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.599583 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.651279 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.651253 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mrs\" (UniqueName: \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\") pod \"maas-api-revoke-sub-79805ca9-skdbq\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:50.783249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:50.783176 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:51.156676 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:51.156644 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq"] Nov 02 07:07:51.163743 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:07:51.163716 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20de29b6_4912_4732_9ce0_a4c76f63b881.slice/crio-8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b WatchSource:0}: Error finding container 8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b: Status 404 returned error can't find the container with id 8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b Nov 02 07:07:51.962311 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:51.962281 6273 generic.go:356] "Generic (PLEG): container finished" podID="20de29b6-4912-4732-9ce0-a4c76f63b881" containerID="8f25619c834db959b0aa841f1e1255df5d012176905564d4a70f686339653f18" exitCode=0 Nov 02 07:07:51.962636 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:51.962355 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" event={"ID":"20de29b6-4912-4732-9ce0-a4c76f63b881","Type":"ContainerDied","Data":"8f25619c834db959b0aa841f1e1255df5d012176905564d4a70f686339653f18"} Nov 02 07:07:51.962636 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:51.962407 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" event={"ID":"20de29b6-4912-4732-9ce0-a4c76f63b881","Type":"ContainerStarted","Data":"8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b"} Nov 02 07:07:53.100600 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.100577 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:53.116945 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.116919 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\") pod \"20de29b6-4912-4732-9ce0-a4c76f63b881\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " Nov 02 07:07:53.117074 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.117025 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\") pod \"20de29b6-4912-4732-9ce0-a4c76f63b881\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " Nov 02 07:07:53.117074 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.117069 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\" (UniqueName: \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\") pod \"20de29b6-4912-4732-9ce0-a4c76f63b881\" (UID: \"20de29b6-4912-4732-9ce0-a4c76f63b881\") " Nov 02 07:07:53.117375 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.117352 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca" pod "20de29b6-4912-4732-9ce0-a4c76f63b881" (UID: "20de29b6-4912-4732-9ce0-a4c76f63b881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:07:53.118915 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.118887 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca" pod "20de29b6-4912-4732-9ce0-a4c76f63b881" (UID: "20de29b6-4912-4732-9ce0-a4c76f63b881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:07:53.119039 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.118949 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs" pod "20de29b6-4912-4732-9ce0-a4c76f63b881" (UID: "20de29b6-4912-4732-9ce0-a4c76f63b881"). InnerVolumeSpecName "kube-api-access-f2mrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:07:53.218056 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.217809 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:53.218056 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.217841 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mrs\" (UniqueName: \"kubernetes.io/projected/20de29b6-4912-4732-9ce0-a4c76f63b881-kube-api-access-f2mrs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:53.218056 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.217857 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de29b6-4912-4732-9ce0-a4c76f63b881-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:07:53.972980 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.972951 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" Nov 02 07:07:53.972980 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.972951 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq" event={"ID":"20de29b6-4912-4732-9ce0-a4c76f63b881","Type":"ContainerDied","Data":"8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b"} Nov 02 07:07:53.973224 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:07:53.973039 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8094c99b84236dfefd0c1d9e138bbca0bce35a54e4d1e585cf600a73306b010b" Nov 02 07:08:04.842816 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:04.842782 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:08:08.552556 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:08:08.552517 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:08:11.063159 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.063124 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr"] Nov 02 07:08:11.063742 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.063721 6273 cpu_manager.go:414] "containerMap: removing container" podUID="20de29b6-4912-4732-9ce0-a4c76f63b881" containerName="revoke-keys" Nov 02 07:08:11.063826 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.063744 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20de29b6-4912-4732-9ce0-a4c76f63b881" containerName="revoke-keys" Nov 02 07:08:11.063920 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.063908 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="20de29b6-4912-4732-9ce0-a4c76f63b881" containerName="revoke-keys" Nov 02 07:08:11.067601 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.067578 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.120717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.120686 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr"] Nov 02 07:08:11.164973 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.164947 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:08:11.164973 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.164946 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:08:11.164973 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.164956 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:08:11.165193 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.165038 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:08:11.225924 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.225895 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56d2c\" (UniqueName: \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.226076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.225948 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.226121 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.226071 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.327288 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.327217 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56d2c\" (UniqueName: \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.327288 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.327268 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.327288 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.327287 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.327948 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.327929 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.329694 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.329676 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.381104 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.381078 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56d2c\" (UniqueName: \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\") pod \"maas-api-revoke-sub-0b7e8edb-wmzxr\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:11.680648 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:11.680618 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:12.051129 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:12.051050 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr"] Nov 02 07:08:12.058235 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:08:12.058205 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b175837_10c4_4441_8507_582ffa984b14.slice/crio-51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce WatchSource:0}: Error finding container 51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce: Status 404 returned error can't find the container with id 51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce Nov 02 07:08:13.070062 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:13.070031 6273 generic.go:356] "Generic (PLEG): container finished" podID="5b175837-10c4-4441-8507-582ffa984b14" containerID="7caaab869d50da8628a606c4ab0ed69974d26b0dab3068c5612839c97bf73455" exitCode=0 Nov 02 07:08:13.070436 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:13.070117 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" event={"ID":"5b175837-10c4-4441-8507-582ffa984b14","Type":"ContainerDied","Data":"7caaab869d50da8628a606c4ab0ed69974d26b0dab3068c5612839c97bf73455"} Nov 02 07:08:13.070436 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:13.070175 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" event={"ID":"5b175837-10c4-4441-8507-582ffa984b14","Type":"ContainerStarted","Data":"51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce"} Nov 02 07:08:14.214249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.214224 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:14.353877 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.353802 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\" (UniqueName: \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\") pod \"5b175837-10c4-4441-8507-582ffa984b14\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " Nov 02 07:08:14.353877 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.353846 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\") pod \"5b175837-10c4-4441-8507-582ffa984b14\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " Nov 02 07:08:14.353877 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.353877 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\") pod \"5b175837-10c4-4441-8507-582ffa984b14\" (UID: \"5b175837-10c4-4441-8507-582ffa984b14\") " Nov 02 07:08:14.354270 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.354245 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca" pod "5b175837-10c4-4441-8507-582ffa984b14" (UID: "5b175837-10c4-4441-8507-582ffa984b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:08:14.354849 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.354820 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b175837-10c4-4441-8507-582ffa984b14-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:14.356152 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.356129 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca" pod "5b175837-10c4-4441-8507-582ffa984b14" (UID: "5b175837-10c4-4441-8507-582ffa984b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:08:14.356244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.356147 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c" pod "5b175837-10c4-4441-8507-582ffa984b14" (UID: "5b175837-10c4-4441-8507-582ffa984b14"). InnerVolumeSpecName "kube-api-access-56d2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:08:14.455560 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.455531 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56d2c\" (UniqueName: \"kubernetes.io/projected/5b175837-10c4-4441-8507-582ffa984b14-kube-api-access-56d2c\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:14.455670 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:14.455563 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b175837-10c4-4441-8507-582ffa984b14-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:15.081138 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:15.081104 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" Nov 02 07:08:15.081378 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:15.081144 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr" event={"ID":"5b175837-10c4-4441-8507-582ffa984b14","Type":"ContainerDied","Data":"51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce"} Nov 02 07:08:15.081378 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:15.081217 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a0b17412c0e786655c17b20f8d09b39e05300ec9e5694524268e06dbc09bce" Nov 02 07:08:22.392885 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.392848 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp"] Nov 02 07:08:22.393337 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.393314 6273 cpu_manager.go:414] "containerMap: removing container" podUID="5b175837-10c4-4441-8507-582ffa984b14" containerName="revoke-keys" Nov 02 07:08:22.393337 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.393326 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b175837-10c4-4441-8507-582ffa984b14" containerName="revoke-keys" Nov 02 07:08:22.393420 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.393402 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b175837-10c4-4441-8507-582ffa984b14" containerName="revoke-keys" Nov 02 07:08:22.396895 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.396869 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.436524 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.436492 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdg2\" (UniqueName: \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.436660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.436631 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.436705 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.436661 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.446106 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.446081 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:08:22.492222 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.492188 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:08:22.492378 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.492246 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:08:22.492378 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.492281 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:08:22.494234 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.494211 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp"] Nov 02 07:08:22.537624 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.537596 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.537624 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.537627 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.537835 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.537670 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdg2\" (UniqueName: \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.538362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.538339 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.540089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.540066 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.591561 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.591532 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdg2\" (UniqueName: \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\") pod \"maas-api-revoke-sub-3d4156f1-czpnp\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:22.709981 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:22.709952 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:23.086171 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:23.086091 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp"] Nov 02 07:08:23.093400 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:08:23.093374 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c94232_cd91_47db_ba8c_f839658fcc5a.slice/crio-04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234 WatchSource:0}: Error finding container 04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234: Status 404 returned error can't find the container with id 04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234 Nov 02 07:08:23.123500 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:23.123469 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" event={"ID":"67c94232-cd91-47db-ba8c-f839658fcc5a","Type":"ContainerStarted","Data":"04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234"} Nov 02 07:08:23.882490 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:23.882455 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:08:24.129581 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:24.129542 6273 generic.go:356] "Generic (PLEG): container finished" podID="67c94232-cd91-47db-ba8c-f839658fcc5a" containerID="555aa0654eba454667561f5baa434009f76054c71451f7a5af63f702c5f733a2" exitCode=0 Nov 02 07:08:24.129751 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:24.129632 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" event={"ID":"67c94232-cd91-47db-ba8c-f839658fcc5a","Type":"ContainerDied","Data":"555aa0654eba454667561f5baa434009f76054c71451f7a5af63f702c5f733a2"} Nov 02 07:08:25.534472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.534440 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:25.621663 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.621620 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\") pod \"67c94232-cd91-47db-ba8c-f839658fcc5a\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " Nov 02 07:08:25.621947 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.621925 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\" (UniqueName: \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\") pod \"67c94232-cd91-47db-ba8c-f839658fcc5a\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " Nov 02 07:08:25.622195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.622176 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\") pod \"67c94232-cd91-47db-ba8c-f839658fcc5a\" (UID: \"67c94232-cd91-47db-ba8c-f839658fcc5a\") " Nov 02 07:08:25.623082 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.623054 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca" pod "67c94232-cd91-47db-ba8c-f839658fcc5a" (UID: "67c94232-cd91-47db-ba8c-f839658fcc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:08:25.626403 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.626369 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca" pod "67c94232-cd91-47db-ba8c-f839658fcc5a" (UID: "67c94232-cd91-47db-ba8c-f839658fcc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:08:25.627345 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.627318 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2" pod "67c94232-cd91-47db-ba8c-f839658fcc5a" (UID: "67c94232-cd91-47db-ba8c-f839658fcc5a"). InnerVolumeSpecName "kube-api-access-tsdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:08:25.730170 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.728896 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:25.730170 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.728943 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdg2\" (UniqueName: \"kubernetes.io/projected/67c94232-cd91-47db-ba8c-f839658fcc5a-kube-api-access-tsdg2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:25.730170 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:25.728960 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c94232-cd91-47db-ba8c-f839658fcc5a-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:26.141325 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:26.141244 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" Nov 02 07:08:26.141489 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:26.141244 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp" event={"ID":"67c94232-cd91-47db-ba8c-f839658fcc5a","Type":"ContainerDied","Data":"04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234"} Nov 02 07:08:26.141489 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:26.141386 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04943404cd16637124cdee8d7b16511dceb3bc91bf6db68438d30aeaf9189234" Nov 02 07:08:36.841192 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:36.841114 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:08:38.612373 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:08:38.612255 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:08:44.056535 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:44.056497 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl"] Nov 02 07:08:44.103666 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:44.099743 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jbcdl"] Nov 02 07:08:44.189145 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:44.189110 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b80116-9cb7-4e0b-8b1e-245a00f9fd46" path="/var/lib/kubelet/pods/c5b80116-9cb7-4e0b-8b1e-245a00f9fd46/volumes" Nov 02 07:08:47.165578 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:47.165545 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:08:52.840508 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.840467 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp"] Nov 02 07:08:52.886710 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.886675 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-czpnp"] Nov 02 07:08:52.955672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.955581 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh"] Nov 02 07:08:52.956127 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.956108 6273 cpu_manager.go:414] "containerMap: removing container" podUID="67c94232-cd91-47db-ba8c-f839658fcc5a" containerName="revoke-keys" Nov 02 07:08:52.956239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.956128 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c94232-cd91-47db-ba8c-f839658fcc5a" containerName="revoke-keys" Nov 02 07:08:52.956301 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.956276 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c94232-cd91-47db-ba8c-f839658fcc5a" containerName="revoke-keys" Nov 02 07:08:52.959431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:52.959410 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.008572 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.008541 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:08:53.008572 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.008560 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:08:53.008908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.008892 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:08:53.009098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.009079 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:08:53.014505 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.014484 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh"] Nov 02 07:08:53.138302 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.138264 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.138493 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.138422 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.138493 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.138462 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsvk\" (UniqueName: \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.239588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.239508 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.239588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.239556 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsvk\" (UniqueName: \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.239787 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.239611 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.240324 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.240298 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.241785 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.241764 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.294733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.294699 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsvk\" (UniqueName: \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\") pod \"maas-api-revoke-sub-3d4156f1-fwglh\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.572158 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.572085 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:53.943104 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:53.943075 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh"] Nov 02 07:08:53.950457 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:08:53.950424 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4be54a_016b_498b_9494_690605233ade.slice/crio-296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548 WatchSource:0}: Error finding container 296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548: Status 404 returned error can't find the container with id 296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548 Nov 02 07:08:54.177875 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:54.177849 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c94232-cd91-47db-ba8c-f839658fcc5a" path="/var/lib/kubelet/pods/67c94232-cd91-47db-ba8c-f839658fcc5a/volumes" Nov 02 07:08:54.302212 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:54.302179 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" event={"ID":"3f4be54a-016b-498b-9494-690605233ade","Type":"ContainerStarted","Data":"e2a3fdc56c614504363b3d694daf9ec0311a728750a777a02c90272c680d55a5"} Nov 02 07:08:54.302348 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:54.302228 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" event={"ID":"3f4be54a-016b-498b-9494-690605233ade","Type":"ContainerStarted","Data":"296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548"} Nov 02 07:08:54.410145 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:54.410088 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" podStartSLOduration=2.410075687 podStartE2EDuration="2.410075687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:08:54.40947918 +0000 UTC m=+3088.382813864" watchObservedRunningTime="2026-11-02 07:08:54.410075687 +0000 UTC m=+3088.383410371" Nov 02 07:08:55.308457 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:55.308422 6273 generic.go:356] "Generic (PLEG): container finished" podID="3f4be54a-016b-498b-9494-690605233ade" containerID="e2a3fdc56c614504363b3d694daf9ec0311a728750a777a02c90272c680d55a5" exitCode=0 Nov 02 07:08:55.308848 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:55.308504 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" event={"ID":"3f4be54a-016b-498b-9494-690605233ade","Type":"ContainerDied","Data":"e2a3fdc56c614504363b3d694daf9ec0311a728750a777a02c90272c680d55a5"} Nov 02 07:08:56.450736 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.450710 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:56.571781 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.571712 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\") pod \"3f4be54a-016b-498b-9494-690605233ade\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " Nov 02 07:08:56.571921 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.571844 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\" (UniqueName: \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\") pod \"3f4be54a-016b-498b-9494-690605233ade\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " Nov 02 07:08:56.571921 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.571878 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\") pod \"3f4be54a-016b-498b-9494-690605233ade\" (UID: \"3f4be54a-016b-498b-9494-690605233ade\") " Nov 02 07:08:56.572152 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.572126 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca" pod "3f4be54a-016b-498b-9494-690605233ade" (UID: "3f4be54a-016b-498b-9494-690605233ade"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:08:56.572262 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.572240 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4be54a-016b-498b-9494-690605233ade-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:56.574020 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.573970 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk" pod "3f4be54a-016b-498b-9494-690605233ade" (UID: "3f4be54a-016b-498b-9494-690605233ade"). InnerVolumeSpecName "kube-api-access-8wsvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:08:56.574129 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.574038 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca" pod "3f4be54a-016b-498b-9494-690605233ade" (UID: "3f4be54a-016b-498b-9494-690605233ade"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:08:56.673662 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.673631 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsvk\" (UniqueName: \"kubernetes.io/projected/3f4be54a-016b-498b-9494-690605233ade-kube-api-access-8wsvk\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:56.673662 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:56.673659 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4be54a-016b-498b-9494-690605233ade-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:08:57.320235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:57.320200 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" Nov 02 07:08:57.320235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:57.320219 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh" event={"ID":"3f4be54a-016b-498b-9494-690605233ade","Type":"ContainerDied","Data":"296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548"} Nov 02 07:08:57.320444 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:08:57.320260 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296c885070081d454e0c62fa06ec5c3d2233951a7f2973ec102c49417c4fe548" Nov 02 07:09:07.462954 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:07.462922 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:09:08.552090 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:09:08.552052 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:09:17.160005 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:17.159954 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:09:31.284029 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:31.283974 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:09:38.552914 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:09:38.552870 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:09:40.078923 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.078886 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm"] Nov 02 07:09:40.079339 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.079325 6273 cpu_manager.go:414] "containerMap: removing container" podUID="3f4be54a-016b-498b-9494-690605233ade" containerName="revoke-keys" Nov 02 07:09:40.079339 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.079337 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4be54a-016b-498b-9494-690605233ade" containerName="revoke-keys" Nov 02 07:09:40.079466 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.079447 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4be54a-016b-498b-9494-690605233ade" containerName="revoke-keys" Nov 02 07:09:40.083975 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.083953 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.180874 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.180844 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:09:40.181058 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.180848 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:09:40.181058 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.180888 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:09:40.181058 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.180887 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:09:40.182818 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.182798 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm"] Nov 02 07:09:40.186271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.186253 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9c2\" (UniqueName: \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.186333 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.186285 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.186450 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.186427 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.287954 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.287916 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.288163 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.288077 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9c2\" (UniqueName: \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.288163 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.288108 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.288740 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.288716 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.290343 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.290317 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.341560 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.341506 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9c2\" (UniqueName: \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\") pod \"maas-api-revoke-sub-e2fc9c5b-wwrwm\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.397511 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.397482 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:40.766170 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:40.766130 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm"] Nov 02 07:09:40.772894 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:09:40.772865 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ff7ac6_ab05_4061_9f5b_552087ed605e.slice/crio-1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3 WatchSource:0}: Error finding container 1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3: Status 404 returned error can't find the container with id 1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3 Nov 02 07:09:41.569176 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:41.569145 6273 generic.go:356] "Generic (PLEG): container finished" podID="37ff7ac6-ab05-4061-9f5b-552087ed605e" containerID="deefbaadcbbc30f245a7bb640f81dc37aee75d50cb4c3eb38544df7ce60f46eb" exitCode=0 Nov 02 07:09:41.569511 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:41.569233 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" event={"ID":"37ff7ac6-ab05-4061-9f5b-552087ed605e","Type":"ContainerDied","Data":"deefbaadcbbc30f245a7bb640f81dc37aee75d50cb4c3eb38544df7ce60f46eb"} Nov 02 07:09:41.569511 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:41.569288 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" event={"ID":"37ff7ac6-ab05-4061-9f5b-552087ed605e","Type":"ContainerStarted","Data":"1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3"} Nov 02 07:09:42.712821 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.712789 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:42.808089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.808061 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\") pod \"37ff7ac6-ab05-4061-9f5b-552087ed605e\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " Nov 02 07:09:42.808240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.808121 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\" (UniqueName: \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\") pod \"37ff7ac6-ab05-4061-9f5b-552087ed605e\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " Nov 02 07:09:42.808240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.808149 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\") pod \"37ff7ac6-ab05-4061-9f5b-552087ed605e\" (UID: \"37ff7ac6-ab05-4061-9f5b-552087ed605e\") " Nov 02 07:09:42.808396 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.808372 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca" pod "37ff7ac6-ab05-4061-9f5b-552087ed605e" (UID: "37ff7ac6-ab05-4061-9f5b-552087ed605e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:09:42.808578 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.808558 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:42.810145 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.810125 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2" pod "37ff7ac6-ab05-4061-9f5b-552087ed605e" (UID: "37ff7ac6-ab05-4061-9f5b-552087ed605e"). InnerVolumeSpecName "kube-api-access-zn9c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:09:42.810197 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.810177 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca" pod "37ff7ac6-ab05-4061-9f5b-552087ed605e" (UID: "37ff7ac6-ab05-4061-9f5b-552087ed605e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:09:42.909796 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.909764 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9c2\" (UniqueName: \"kubernetes.io/projected/37ff7ac6-ab05-4061-9f5b-552087ed605e-kube-api-access-zn9c2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:42.909796 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:42.909793 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ff7ac6-ab05-4061-9f5b-552087ed605e-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:43.581355 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:43.581327 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" event={"ID":"37ff7ac6-ab05-4061-9f5b-552087ed605e","Type":"ContainerDied","Data":"1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3"} Nov 02 07:09:43.581529 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:43.581360 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm" Nov 02 07:09:43.581529 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:43.581373 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d2ab3fc59d25aa8740859e6e8ae3c2ac0ccd2b3f09b2c09cb59556d1098e1a3" Nov 02 07:09:50.813774 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.813738 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t"] Nov 02 07:09:50.814195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.814182 6273 cpu_manager.go:414] "containerMap: removing container" podUID="37ff7ac6-ab05-4061-9f5b-552087ed605e" containerName="revoke-keys" Nov 02 07:09:50.814246 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.814195 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ff7ac6-ab05-4061-9f5b-552087ed605e" containerName="revoke-keys" Nov 02 07:09:50.814304 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.814293 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ff7ac6-ab05-4061-9f5b-552087ed605e" containerName="revoke-keys" Nov 02 07:09:50.817733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.817706 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.885918 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.885884 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlh2\" (UniqueName: \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.886095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.886016 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.886095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.886060 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.915647 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.915623 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:09:50.915802 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.915687 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:09:50.915802 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.915719 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:09:50.915943 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.915904 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:09:50.918740 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.918715 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t"] Nov 02 07:09:50.987095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.987069 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlh2\" (UniqueName: \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.987230 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.987122 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.987314 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.987296 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.987826 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.987801 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:50.989678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:50.989657 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:51.040614 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:51.040594 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlh2\" (UniqueName: \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\") pod \"maas-api-revoke-sub-5a29efb3-mzk7t\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:51.131136 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:51.131074 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:51.501279 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:51.501246 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t"] Nov 02 07:09:51.508000 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:09:51.507963 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be54940_9886_4aac_aa48_84f6756a658e.slice/crio-76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597 WatchSource:0}: Error finding container 76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597: Status 404 returned error can't find the container with id 76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597 Nov 02 07:09:51.626551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:51.626520 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" event={"ID":"2be54940-9886-4aac-aa48-84f6756a658e","Type":"ContainerStarted","Data":"76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597"} Nov 02 07:09:52.055951 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:52.055882 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9"] Nov 02 07:09:52.060738 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:52.060715 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6vbc9"] Nov 02 07:09:52.178665 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:52.178628 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10c58bf-ba92-4fab-8e05-7db00d148982" path="/var/lib/kubelet/pods/f10c58bf-ba92-4fab-8e05-7db00d148982/volumes" Nov 02 07:09:52.632870 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:52.632766 6273 generic.go:356] "Generic (PLEG): container finished" podID="2be54940-9886-4aac-aa48-84f6756a658e" containerID="68989d93c814033cfd19cbb62d96ac15d747cbcf57956ae6c3e31d7f14c5a3cd" exitCode=0 Nov 02 07:09:52.632870 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:52.632842 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" event={"ID":"2be54940-9886-4aac-aa48-84f6756a658e","Type":"ContainerDied","Data":"68989d93c814033cfd19cbb62d96ac15d747cbcf57956ae6c3e31d7f14c5a3cd"} Nov 02 07:09:53.003978 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.003944 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:09:53.804228 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.804196 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:09:53.914094 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.914050 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\") pod \"2be54940-9886-4aac-aa48-84f6756a658e\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " Nov 02 07:09:53.914291 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.914104 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\" (UniqueName: \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\") pod \"2be54940-9886-4aac-aa48-84f6756a658e\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " Nov 02 07:09:53.914291 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.914227 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\") pod \"2be54940-9886-4aac-aa48-84f6756a658e\" (UID: \"2be54940-9886-4aac-aa48-84f6756a658e\") " Nov 02 07:09:53.914925 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.914820 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca" pod "2be54940-9886-4aac-aa48-84f6756a658e" (UID: "2be54940-9886-4aac-aa48-84f6756a658e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:09:53.918738 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.918708 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2" pod "2be54940-9886-4aac-aa48-84f6756a658e" (UID: "2be54940-9886-4aac-aa48-84f6756a658e"). InnerVolumeSpecName "kube-api-access-rtlh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:09:53.921044 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:53.920982 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca" pod "2be54940-9886-4aac-aa48-84f6756a658e" (UID: "2be54940-9886-4aac-aa48-84f6756a658e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:09:54.015342 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.015310 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:54.015342 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.015336 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlh2\" (UniqueName: \"kubernetes.io/projected/2be54940-9886-4aac-aa48-84f6756a658e-kube-api-access-rtlh2\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:54.015342 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.015347 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be54940-9886-4aac-aa48-84f6756a658e-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:09:54.673018 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.672697 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" event={"ID":"2be54940-9886-4aac-aa48-84f6756a658e","Type":"ContainerDied","Data":"76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597"} Nov 02 07:09:54.673018 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.672762 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cbb847f198ff3ef0cc9f96190fd1856cb140e464d5e5ac5409ed309e5ba597" Nov 02 07:09:54.673018 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:09:54.672846 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t" Nov 02 07:10:03.575775 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:03.575699 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:10:07.734210 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.734176 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh"] Nov 02 07:10:07.780461 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.780430 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwglh"] Nov 02 07:10:07.848351 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.848319 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz"] Nov 02 07:10:07.848800 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.848779 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2be54940-9886-4aac-aa48-84f6756a658e" containerName="revoke-keys" Nov 02 07:10:07.848861 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.848803 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be54940-9886-4aac-aa48-84f6756a658e" containerName="revoke-keys" Nov 02 07:10:07.848902 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.848880 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be54940-9886-4aac-aa48-84f6756a658e" containerName="revoke-keys" Nov 02 07:10:07.852212 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.852195 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:07.903974 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.903938 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:10:07.903974 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.903945 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:10:07.904197 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.904060 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:10:07.904197 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.904062 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:10:07.905933 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.905907 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz"] Nov 02 07:10:07.947430 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.947404 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7hj\" (UniqueName: \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:07.947564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.947448 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:07.947564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:07.947466 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.048549 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.048480 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7hj\" (UniqueName: \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.048549 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.048528 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.048549 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.048547 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.049249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.049229 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.050827 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.050808 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.101966 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.101940 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7hj\" (UniqueName: \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\") pod \"maas-api-revoke-sub-3d4156f1-962zz\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.165711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.165684 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:08.178559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.178529 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4be54a-016b-498b-9494-690605233ade" path="/var/lib/kubelet/pods/3f4be54a-016b-498b-9494-690605233ade/volumes" Nov 02 07:10:08.539506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.539474 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz"] Nov 02 07:10:08.546344 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:10:08.546318 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0639c_4aa2_44f7_98cb_b2f6530a5b58.slice/crio-03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86 WatchSource:0}: Error finding container 03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86: Status 404 returned error can't find the container with id 03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86 Nov 02 07:10:08.555086 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:10:08.555044 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:10:08.758138 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:08.758106 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" event={"ID":"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58","Type":"ContainerStarted","Data":"03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86"} Nov 02 07:10:09.764980 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:09.764949 6273 generic.go:356] "Generic (PLEG): container finished" podID="e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" containerID="c0e1d31da45460ae243704f0525075c208e7a9a1a787c885fd3db246359c56bb" exitCode=0 Nov 02 07:10:09.765422 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:09.765035 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" event={"ID":"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58","Type":"ContainerDied","Data":"c0e1d31da45460ae243704f0525075c208e7a9a1a787c885fd3db246359c56bb"} Nov 02 07:10:10.912066 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.912045 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:10.976525 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.976500 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\") pod \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " Nov 02 07:10:10.976641 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.976555 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\") pod \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " Nov 02 07:10:10.976641 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.976611 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\" (UniqueName: \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\") pod \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\" (UID: \"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58\") " Nov 02 07:10:10.976886 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.976859 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca" pod "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" (UID: "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:10:10.978499 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.978475 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca" pod "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" (UID: "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:10:10.978676 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:10.978656 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj" pod "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" (UID: "e9d0639c-4aa2-44f7-98cb-b2f6530a5b58"). InnerVolumeSpecName "kube-api-access-5n7hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:10:11.077844 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.077779 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:10:11.077844 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.077803 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:10:11.077844 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.077813 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7hj\" (UniqueName: \"kubernetes.io/projected/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58-kube-api-access-5n7hj\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:10:11.777076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.777040 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" event={"ID":"e9d0639c-4aa2-44f7-98cb-b2f6530a5b58","Type":"ContainerDied","Data":"03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86"} Nov 02 07:10:11.777244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.777080 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz" Nov 02 07:10:11.777244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:11.777112 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ce977828de63b9c26f6a31afd3ac2086084323a3c7fac87bb3feb7796efa86" Nov 02 07:10:22.370117 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:22.370081 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:10:23.108037 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:23.096366 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9"] Nov 02 07:10:23.132165 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:23.132129 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vgbm9"] Nov 02 07:10:24.198014 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:24.197280 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263f3a9a-9ceb-4e4b-9981-c0c706784795" path="/var/lib/kubelet/pods/263f3a9a-9ceb-4e4b-9981-c0c706784795/volumes" Nov 02 07:10:27.533953 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:27.533918 6273 scope.go:122] "RemoveContainer" containerID="9db322451ec7290fe97b08c15a7bc825e7ad59ea11b59e2539ef334e72cef96c" Nov 02 07:10:32.341267 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:32.341231 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:10:35.044133 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:35.044098 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c"] Nov 02 07:10:35.090710 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:35.090681 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h22c"] Nov 02 07:10:36.178614 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:36.178580 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1525379-ad0c-4e8d-ac63-33f2ba823694" path="/var/lib/kubelet/pods/d1525379-ad0c-4e8d-ac63-33f2ba823694/volumes" Nov 02 07:10:38.552201 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:10:38.552159 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:10:47.805760 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.805731 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:10:47.806179 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.806146 6273 cpu_manager.go:414] "containerMap: removing container" podUID="e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" containerName="revoke-keys" Nov 02 07:10:47.806179 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.806158 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" containerName="revoke-keys" Nov 02 07:10:47.806261 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.806254 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" containerName="revoke-keys" Nov 02 07:10:47.816845 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.816816 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:47.863655 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.863624 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:10:47.908237 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.908202 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-614951\"" Nov 02 07:10:47.908237 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.908202 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-650de355-dockercfg-j6wh8\"" Nov 02 07:10:47.951096 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.951063 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:47.951265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.951204 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5wp\" (UniqueName: \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:47.968232 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.968206 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:10:47.972037 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:47.972011 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.025984 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.025949 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:10:48.052249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.052218 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5wp\" (UniqueName: \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.052435 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.052300 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72xx\" (UniqueName: \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.052435 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.052354 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.052559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.052470 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.053125 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.053102 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.106431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.106353 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5wp\" (UniqueName: \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\") pod \"payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.131416 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.131383 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.153422 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.153381 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72xx\" (UniqueName: \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.153581 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.153487 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.154276 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.154252 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.208538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.208505 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72xx\" (UniqueName: \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\") pod \"payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.285739 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.285706 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:48.510223 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.510190 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:10:48.516630 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:10:48.516604 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61149048_1e88_4b5f_ba34_07522f200893.slice/crio-24dd363a318ea853298b66bbe56dee18479972a9663a4eccc165598bdff45da8 WatchSource:0}: Error finding container 24dd363a318ea853298b66bbe56dee18479972a9663a4eccc165598bdff45da8: Status 404 returned error can't find the container with id 24dd363a318ea853298b66bbe56dee18479972a9663a4eccc165598bdff45da8 Nov 02 07:10:48.660306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.660275 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:10:48.667557 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:10:48.667525 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975e62b9_53b0_4b81_8814_c1abb06b4d87.slice/crio-1e2d44236acaf138862ae7b6b01b49a7f1d6162094fc7aefb5ffd0dcf21cd662 WatchSource:0}: Error finding container 1e2d44236acaf138862ae7b6b01b49a7f1d6162094fc7aefb5ffd0dcf21cd662: Status 404 returned error can't find the container with id 1e2d44236acaf138862ae7b6b01b49a7f1d6162094fc7aefb5ffd0dcf21cd662 Nov 02 07:10:48.974215 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.974180 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" event={"ID":"61149048-1e88-4b5f-ba34-07522f200893","Type":"ContainerStarted","Data":"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c"} Nov 02 07:10:48.974744 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.974248 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" event={"ID":"61149048-1e88-4b5f-ba34-07522f200893","Type":"ContainerStarted","Data":"24dd363a318ea853298b66bbe56dee18479972a9663a4eccc165598bdff45da8"} Nov 02 07:10:48.974744 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.974364 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:48.975864 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.975832 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" event={"ID":"975e62b9-53b0-4b81-8814-c1abb06b4d87","Type":"ContainerStarted","Data":"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4"} Nov 02 07:10:48.975974 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.975878 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" event={"ID":"975e62b9-53b0-4b81-8814-c1abb06b4d87","Type":"ContainerStarted","Data":"1e2d44236acaf138862ae7b6b01b49a7f1d6162094fc7aefb5ffd0dcf21cd662"} Nov 02 07:10:48.975974 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:48.975904 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:10:49.082745 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:49.082691 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" podStartSLOduration=2.08267527 podStartE2EDuration="2.08267527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:10:49.081962642 +0000 UTC m=+3203.055297326" watchObservedRunningTime="2026-11-02 07:10:49.08267527 +0000 UTC m=+3203.056009981" Nov 02 07:10:49.191325 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:49.191112 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" podStartSLOduration=2.191095692 podStartE2EDuration="2.191095692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:10:49.189791752 +0000 UTC m=+3203.163126436" watchObservedRunningTime="2026-11-02 07:10:49.191095692 +0000 UTC m=+3203.164430377" Nov 02 07:10:50.012241 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.012200 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:10:50.016232 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.016212 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.073284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.073248 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.073458 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.073374 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.073523 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.073470 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7n7h\" (UniqueName: \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.073777 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.073755 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.112099 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.112063 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:10:50.112099 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.112096 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-614951-serving-cert\"" Nov 02 07:10:50.112294 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.112280 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 07:10:50.112339 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.112306 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:10:50.114037 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.114018 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:10:50.115197 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.115184 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 07:10:50.174408 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.174372 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7n7h\" (UniqueName: \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.174591 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.174462 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.174591 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.174552 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.174684 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.174593 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.174725 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:10:50.174706 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-serving-cert: secret "maas-api-e2e-worker-single-main-614951-serving-cert" not found Nov 02 07:10:50.174776 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:10:50.174765 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls podName:d64c558f-5a36-4308-883d-86df2ed57a72 nodeName:}" failed. No retries permitted until 2026-11-02 07:10:50.674746834 +0000 UTC m=+3204.648081511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls") pod "maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" (UID: "d64c558f-5a36-4308-883d-86df2ed57a72") : secret "maas-api-e2e-worker-single-main-614951-serving-cert" not found Nov 02 07:10:50.175240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.175220 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.177109 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.177088 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.228841 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.228810 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7n7h\" (UniqueName: \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.679764 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.679724 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.682330 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.682300 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:50.930506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:50.930423 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:51.301338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:51.301264 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:10:51.307527 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:10:51.307496 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64c558f_5a36_4308_883d_86df2ed57a72.slice/crio-176b73898270c70b6fa457239e620d6191110310599210cd328a886e8cf38d2e WatchSource:0}: Error finding container 176b73898270c70b6fa457239e620d6191110310599210cd328a886e8cf38d2e: Status 404 returned error can't find the container with id 176b73898270c70b6fa457239e620d6191110310599210cd328a886e8cf38d2e Nov 02 07:10:51.309172 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:51.309152 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:10:52.003070 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:52.003027 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" event={"ID":"d64c558f-5a36-4308-883d-86df2ed57a72","Type":"ContainerStarted","Data":"176b73898270c70b6fa457239e620d6191110310599210cd328a886e8cf38d2e"} Nov 02 07:10:53.011057 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:53.010950 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" event={"ID":"d64c558f-5a36-4308-883d-86df2ed57a72","Type":"ContainerStarted","Data":"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c"} Nov 02 07:10:53.011057 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:53.011045 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:53.120206 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:53.120148 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" podStartSLOduration=3.167514216 podStartE2EDuration="4.120133679s" totalImagesPullingTime="952.619463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:10:53.118720637 +0000 UTC m=+3207.092055408" watchObservedRunningTime="2026-11-02 07:10:53.120133679 +0000 UTC m=+3207.093468363" Nov 02 07:10:57.788737 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:57.788703 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:10:59.022388 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:59.022355 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:10:59.984725 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:59.984696 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:10:59.984919 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:10:59.984761 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:11:08.553843 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:11:08.553800 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:11:10.049399 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:10.049362 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw"] Nov 02 07:11:10.096031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:10.095984 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xkjw"] Nov 02 07:11:10.178048 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:10.178016 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a725dedc-adfb-4530-8592-71b9b758a7f3" path="/var/lib/kubelet/pods/a725dedc-adfb-4530-8592-71b9b758a7f3/volumes" Nov 02 07:11:12.986575 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:12.986539 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:11:12.991031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:12.990978 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003489 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003464 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003503 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlpl\" (UniqueName: \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003569 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003687 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003612 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003687 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003682 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.003751 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.003723 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.087510 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.087477 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-614951\"/\"simulator-single-main-614951-kserve-self-signed-certs\"" Nov 02 07:11:13.087686 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.087537 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-614951\"/\"default-dockercfg-jhjfg\"" Nov 02 07:11:13.087686 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.087675 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-614951\"/\"kube-root-ca.crt\"" Nov 02 07:11:13.087773 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.087700 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-614951\"/\"openshift-service-ca.crt\"" Nov 02 07:11:13.089660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.089643 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:11:13.104361 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104337 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104427 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104388 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104512 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104495 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104572 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104623 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104606 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlpl\" (UniqueName: \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104667 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104635 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104756 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104738 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.104846 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104827 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.105046 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.104977 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.106639 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.106614 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.106933 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.106914 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.159210 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.159180 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlpl\" (UniqueName: \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\") pod \"simulator-single-main-614951-kserve-799d858786-qmhg6\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.303521 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.303434 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:13.671039 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:13.670980 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:11:13.673452 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:11:13.673424 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f79b09e_0f3c_4485_b97f_cf9f21860110.slice/crio-ff195eb7f311848f0268b70432e51ce9f9303a008f911c7e825b4f17a6895744 WatchSource:0}: Error finding container ff195eb7f311848f0268b70432e51ce9f9303a008f911c7e825b4f17a6895744: Status 404 returned error can't find the container with id ff195eb7f311848f0268b70432e51ce9f9303a008f911c7e825b4f17a6895744 Nov 02 07:11:14.122654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:14.122578 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" event={"ID":"0f79b09e-0f3c-4485-b97f-cf9f21860110","Type":"ContainerStarted","Data":"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f"} Nov 02 07:11:14.122654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:14.122617 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" event={"ID":"0f79b09e-0f3c-4485-b97f-cf9f21860110","Type":"ContainerStarted","Data":"ff195eb7f311848f0268b70432e51ce9f9303a008f911c7e825b4f17a6895744"} Nov 02 07:11:14.231523 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:14.231474 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" podStartSLOduration=2.231459296 podStartE2EDuration="2.231459296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:11:14.22964027 +0000 UTC m=+3228.202974953" watchObservedRunningTime="2026-11-02 07:11:14.231459296 +0000 UTC m=+3228.204793976" Nov 02 07:11:14.304305 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:14.304276 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:17.551334 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:17.551284 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:11:21.045036 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:21.044978 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz"] Nov 02 07:11:21.091478 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:21.091443 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v8zqz"] Nov 02 07:11:22.178669 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:22.178635 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23efa26a-dcaa-423b-a7e4-6ff90f05c06b" path="/var/lib/kubelet/pods/23efa26a-dcaa-423b-a7e4-6ff90f05c06b/volumes" Nov 02 07:11:23.303782 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:23.303742 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:23.316430 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:23.316400 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:24.190357 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:24.190325 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:11:27.561979 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:27.561945 6273 scope.go:122] "RemoveContainer" containerID="8074d6980d27a5a17e97930516ca4719c34a6ffdfe279839c5fd41914b14cb07" Nov 02 07:11:27.573636 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:27.573612 6273 scope.go:122] "RemoveContainer" containerID="9e7b4438c4223ae1b6676a396325e83524a109a0bf1b7510a129da7e087b0115" Nov 02 07:11:38.554032 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:11:38.553937 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:11:42.899967 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:42.899930 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:11:55.049429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:55.049392 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw"] Nov 02 07:11:55.095772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:55.095741 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k4pqw"] Nov 02 07:11:56.182506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:11:56.181805 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cdb369-b36e-494d-8406-5002f1b5cfe0" path="/var/lib/kubelet/pods/32cdb369-b36e-494d-8406-5002f1b5cfe0/volumes" Nov 02 07:12:08.553005 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:08.552954 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:12:11.974448 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:11.974408 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:12:13.805714 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:13.805682 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:12:20.730745 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:20.730710 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:12:20.829307 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:20.829272 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:12:23.044944 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:23.044905 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n"] Nov 02 07:12:23.047717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:23.047689 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hkl4n"] Nov 02 07:12:24.178191 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:24.178159 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5ebea7-99cf-4267-ad9e-048bf79dd6df" path="/var/lib/kubelet/pods/5e5ebea7-99cf-4267-ad9e-048bf79dd6df/volumes" Nov 02 07:12:25.731284 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:25.731226 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" containerID="cri-o://cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4" gracePeriod=25 Nov 02 07:12:25.829973 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:25.829934 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" containerID="cri-o://a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c" gracePeriod=25 Nov 02 07:12:27.598862 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:27.598828 6273 scope.go:122] "RemoveContainer" containerID="36c3ca97dc9fcc1b540ab8450f16f7e10317aea8223cfd912f9a0b953bbb4ce4" Nov 02 07:12:27.610168 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:27.610141 6273 scope.go:122] "RemoveContainer" containerID="eae9439769a908c24090001693eb523362004e6f839c8cd6b739f313012041b3" Nov 02 07:12:27.618564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:27.618544 6273 scope.go:122] "RemoveContainer" containerID="ace0f9c2e98a83ced0506e39eac01688ce3a91367bde6cb743d8d7e307f1d67a" Nov 02 07:12:28.455678 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:28.455644 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:12:28.455868 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:28.455831 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" containerName="maas-api" containerID="cri-o://56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c" gracePeriod=30 Nov 02 07:12:28.825853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:28.825764 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" podUID="0f79b09e-0f3c-4485-b97f-cf9f21860110" containerName="main" containerID="cri-o://cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f" gracePeriod=45 Nov 02 07:12:29.017597 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.017552 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.175:8443/health\": dial tcp 10.134.0.175:8443: connect: connection refused" Nov 02 07:12:29.077182 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.077127 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:12:29.243651 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243621 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.243851 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243672 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.243851 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243702 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\" (UniqueName: \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.243851 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243759 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.243851 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243843 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.244098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243905 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\") pod \"0f79b09e-0f3c-4485-b97f-cf9f21860110\" (UID: \"0f79b09e-0f3c-4485-b97f-cf9f21860110\") " Nov 02 07:12:29.244098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243964 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:29.244098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.243977 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:29.244239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.244177 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:29.244397 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.244365 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.244397 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.244386 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.244397 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.244399 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.246028 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.246007 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "kube-api-access-wmlpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:29.246456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.246435 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:29.246456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.246444 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs" pod "0f79b09e-0f3c-4485-b97f-cf9f21860110" (UID: "0f79b09e-0f3c-4485-b97f-cf9f21860110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:29.345349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.345270 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlpl\" (UniqueName: \"kubernetes.io/projected/0f79b09e-0f3c-4485-b97f-cf9f21860110-kube-api-access-wmlpl\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.345349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.345300 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f79b09e-0f3c-4485-b97f-cf9f21860110-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.345349 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.345311 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f79b09e-0f3c-4485-b97f-cf9f21860110-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.536311 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.536276 6273 generic.go:356] "Generic (PLEG): container finished" podID="0f79b09e-0f3c-4485-b97f-cf9f21860110" containerID="cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f" exitCode=0 Nov 02 07:12:29.536492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.536314 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" Nov 02 07:12:29.536492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.536361 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" event={"ID":"0f79b09e-0f3c-4485-b97f-cf9f21860110","Type":"ContainerDied","Data":"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f"} Nov 02 07:12:29.536492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.536409 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6" event={"ID":"0f79b09e-0f3c-4485-b97f-cf9f21860110","Type":"ContainerDied","Data":"ff195eb7f311848f0268b70432e51ce9f9303a008f911c7e825b4f17a6895744"} Nov 02 07:12:29.536492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.536430 6273 scope.go:122] "RemoveContainer" containerID="cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f" Nov 02 07:12:29.546344 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.546327 6273 scope.go:122] "RemoveContainer" containerID="cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f" Nov 02 07:12:29.546604 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:29.546586 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f\": container with ID starting with cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f not found: ID does not exist" containerID="cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f" Nov 02 07:12:29.546672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.546617 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f"} err="failed to get container status \"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f\": rpc error: code = NotFound desc = could not find container \"cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f\": container with ID starting with cfc7f99a7196b6560faa10674719ef544d5746e54b5bd97ef1755ed25eeab07f not found: ID does not exist" Nov 02 07:12:29.690849 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.690824 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:12:29.692880 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.692856 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:12:29.736972 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.736949 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-614951/simulator-single-main-614951-kserve-799d858786-qmhg6"] Nov 02 07:12:29.850666 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.850642 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") pod \"d64c558f-5a36-4308-883d-86df2ed57a72\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " Nov 02 07:12:29.851068 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.850691 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\" (UniqueName: \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\") pod \"d64c558f-5a36-4308-883d-86df2ed57a72\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " Nov 02 07:12:29.851068 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.850730 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\") pod \"d64c558f-5a36-4308-883d-86df2ed57a72\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " Nov 02 07:12:29.851068 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.850767 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\") pod \"d64c558f-5a36-4308-883d-86df2ed57a72\" (UID: \"d64c558f-5a36-4308-883d-86df2ed57a72\") " Nov 02 07:12:29.851306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.851252 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca" pod "d64c558f-5a36-4308-883d-86df2ed57a72" (UID: "d64c558f-5a36-4308-883d-86df2ed57a72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:29.852779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.852755 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h" pod "d64c558f-5a36-4308-883d-86df2ed57a72" (UID: "d64c558f-5a36-4308-883d-86df2ed57a72"). InnerVolumeSpecName "kube-api-access-q7n7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:29.853199 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.853177 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs" pod "d64c558f-5a36-4308-883d-86df2ed57a72" (UID: "d64c558f-5a36-4308-883d-86df2ed57a72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:29.853278 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.853193 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls" pod "d64c558f-5a36-4308-883d-86df2ed57a72" (UID: "d64c558f-5a36-4308-883d-86df2ed57a72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:29.952558 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.952532 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.952558 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.952556 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7n7h\" (UniqueName: \"kubernetes.io/projected/d64c558f-5a36-4308-883d-86df2ed57a72-kube-api-access-q7n7h\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.952700 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.952567 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64c558f-5a36-4308-883d-86df2ed57a72-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.952700 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.952576 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64c558f-5a36-4308-883d-86df2ed57a72-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:29.983236 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.983207 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:12:29.983329 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:29.983276 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:12:30.178846 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.178814 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f79b09e-0f3c-4485-b97f-cf9f21860110" path="/var/lib/kubelet/pods/0f79b09e-0f3c-4485-b97f-cf9f21860110/volumes" Nov 02 07:12:30.544103 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.544068 6273 generic.go:356] "Generic (PLEG): container finished" podID="d64c558f-5a36-4308-883d-86df2ed57a72" containerID="56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c" exitCode=0 Nov 02 07:12:30.544271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.544117 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" Nov 02 07:12:30.544271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.544131 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" event={"ID":"d64c558f-5a36-4308-883d-86df2ed57a72","Type":"ContainerDied","Data":"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c"} Nov 02 07:12:30.544271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.544175 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7" event={"ID":"d64c558f-5a36-4308-883d-86df2ed57a72","Type":"ContainerDied","Data":"176b73898270c70b6fa457239e620d6191110310599210cd328a886e8cf38d2e"} Nov 02 07:12:30.544271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.544196 6273 scope.go:122] "RemoveContainer" containerID="56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c" Nov 02 07:12:30.553301 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.553282 6273 scope.go:122] "RemoveContainer" containerID="56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c" Nov 02 07:12:30.553588 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:30.553562 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c\": container with ID starting with 56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c not found: ID does not exist" containerID="56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c" Nov 02 07:12:30.553642 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.553599 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c"} err="failed to get container status \"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c\": rpc error: code = NotFound desc = could not find container \"56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c\": container with ID starting with 56d71438d2032bce47a8bbe2f605d366733aba1a1c39feb5fb84e0c9ed17528c not found: ID does not exist" Nov 02 07:12:30.697377 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.697345 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:12:30.743433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:30.743403 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-614951-6557878c5d-tpcc7"] Nov 02 07:12:32.179638 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:32.179599 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" path="/var/lib/kubelet/pods/d64c558f-5a36-4308-883d-86df2ed57a72/volumes" Nov 02 07:12:33.166180 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:33.166151 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:12:38.552465 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:38.552421 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:12:38.617746 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.617718 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz"] Nov 02 07:12:38.664089 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.664059 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-962zz"] Nov 02 07:12:38.727797 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.727771 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw"] Nov 02 07:12:38.728216 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728203 6273 cpu_manager.go:414] "containerMap: removing container" podUID="0f79b09e-0f3c-4485-b97f-cf9f21860110" containerName="main" Nov 02 07:12:38.728275 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728217 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f79b09e-0f3c-4485-b97f-cf9f21860110" containerName="main" Nov 02 07:12:38.728275 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728230 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" containerName="maas-api" Nov 02 07:12:38.728275 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728235 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" containerName="maas-api" Nov 02 07:12:38.728377 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728299 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f79b09e-0f3c-4485-b97f-cf9f21860110" containerName="main" Nov 02 07:12:38.728377 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.728310 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64c558f-5a36-4308-883d-86df2ed57a72" containerName="maas-api" Nov 02 07:12:38.732747 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.732729 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.783831 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.783800 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:12:38.784092 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.784070 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:12:38.785485 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.785462 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw"] Nov 02 07:12:38.828436 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.828372 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.828545 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.828448 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.828545 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.828528 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp27\" (UniqueName: \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.829807 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.829791 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:12:38.829854 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.829796 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:12:38.929465 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.929437 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.929637 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.929480 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp27\" (UniqueName: \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.929637 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.929543 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.930298 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.930274 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.931890 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.931870 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:38.983283 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:38.983259 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp27\" (UniqueName: \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\") pod \"maas-api-revoke-sub-3d4156f1-xlgpw\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:39.045702 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:39.045679 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:39.411638 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:39.411613 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw"] Nov 02 07:12:39.417133 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:12:39.417097 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c065deb_6cff_42c4_aff0_597ed8a6d5d3.slice/crio-9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1 WatchSource:0}: Error finding container 9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1: Status 404 returned error can't find the container with id 9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1 Nov 02 07:12:39.592492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:39.592456 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" event={"ID":"7c065deb-6cff-42c4-aff0-597ed8a6d5d3","Type":"ContainerStarted","Data":"9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1"} Nov 02 07:12:39.983786 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:39.983746 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:12:39.983943 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:39.983752 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:12:40.178196 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:40.178167 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d0639c-4aa2-44f7-98cb-b2f6530a5b58" path="/var/lib/kubelet/pods/e9d0639c-4aa2-44f7-98cb-b2f6530a5b58/volumes" Nov 02 07:12:40.599024 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:40.598921 6273 generic.go:356] "Generic (PLEG): container finished" podID="7c065deb-6cff-42c4-aff0-597ed8a6d5d3" containerID="03131d5e06e150331a1bdf10431314e146b5005d65170de24adc3670da4ca2cf" exitCode=0 Nov 02 07:12:40.599494 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:40.599032 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" event={"ID":"7c065deb-6cff-42c4-aff0-597ed8a6d5d3","Type":"ContainerDied","Data":"03131d5e06e150331a1bdf10431314e146b5005d65170de24adc3670da4ca2cf"} Nov 02 07:12:41.740462 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.740435 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:41.756599 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.756575 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\") pod \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " Nov 02 07:12:41.756731 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.756691 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\" (UniqueName: \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\") pod \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " Nov 02 07:12:41.756731 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.756715 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\") pod \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\" (UID: \"7c065deb-6cff-42c4-aff0-597ed8a6d5d3\") " Nov 02 07:12:41.756964 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.756938 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca" pod "7c065deb-6cff-42c4-aff0-597ed8a6d5d3" (UID: "7c065deb-6cff-42c4-aff0-597ed8a6d5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:41.758749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.758724 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27" pod "7c065deb-6cff-42c4-aff0-597ed8a6d5d3" (UID: "7c065deb-6cff-42c4-aff0-597ed8a6d5d3"). InnerVolumeSpecName "kube-api-access-vgp27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:41.758842 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.758746 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca" pod "7c065deb-6cff-42c4-aff0-597ed8a6d5d3" (UID: "7c065deb-6cff-42c4-aff0-597ed8a6d5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:41.858266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.858187 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp27\" (UniqueName: \"kubernetes.io/projected/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-kube-api-access-vgp27\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:41.858266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.858217 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:41.858266 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:41.858234 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c065deb-6cff-42c4-aff0-597ed8a6d5d3-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.610704 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:42.610631 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" Nov 02 07:12:42.610704 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:42.610663 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw" event={"ID":"7c065deb-6cff-42c4-aff0-597ed8a6d5d3","Type":"ContainerDied","Data":"9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1"} Nov 02 07:12:42.610901 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:42.610712 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9639a5d7ca4c18140bebeef5f9c799efab29e32cdf69a2e579d76da2d0118ac1" Nov 02 07:12:45.988366 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:45.988308 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:12:46.074154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.074127 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:12:46.097496 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.097469 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\" (UniqueName: \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\") pod \"975e62b9-53b0-4b81-8814-c1abb06b4d87\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " Nov 02 07:12:46.097599 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.097536 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\") pod \"975e62b9-53b0-4b81-8814-c1abb06b4d87\" (UID: \"975e62b9-53b0-4b81-8814-c1abb06b4d87\") " Nov 02 07:12:46.097861 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.097838 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume" pod "975e62b9-53b0-4b81-8814-c1abb06b4d87" (UID: "975e62b9-53b0-4b81-8814-c1abb06b4d87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:46.097937 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.097921 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975e62b9-53b0-4b81-8814-c1abb06b4d87-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:46.099448 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.099432 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx" pod "975e62b9-53b0-4b81-8814-c1abb06b4d87" (UID: "975e62b9-53b0-4b81-8814-c1abb06b4d87"). InnerVolumeSpecName "kube-api-access-c72xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:46.198776 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.198751 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\") pod \"61149048-1e88-4b5f-ba34-07522f200893\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " Nov 02 07:12:46.198890 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.198789 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\" (UniqueName: \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\") pod \"61149048-1e88-4b5f-ba34-07522f200893\" (UID: \"61149048-1e88-4b5f-ba34-07522f200893\") " Nov 02 07:12:46.199078 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.199062 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72xx\" (UniqueName: \"kubernetes.io/projected/975e62b9-53b0-4b81-8814-c1abb06b4d87-kube-api-access-c72xx\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:46.199127 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.199094 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume" pod "61149048-1e88-4b5f-ba34-07522f200893" (UID: "61149048-1e88-4b5f-ba34-07522f200893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:46.200739 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.200720 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp" pod "61149048-1e88-4b5f-ba34-07522f200893" (UID: "61149048-1e88-4b5f-ba34-07522f200893"). InnerVolumeSpecName "kube-api-access-wf5wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:46.299669 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.299582 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61149048-1e88-4b5f-ba34-07522f200893-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:46.299669 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.299609 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5wp\" (UniqueName: \"kubernetes.io/projected/61149048-1e88-4b5f-ba34-07522f200893-kube-api-access-wf5wp\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:46.632855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.632756 6273 generic.go:356] "Generic (PLEG): container finished" podID="61149048-1e88-4b5f-ba34-07522f200893" containerID="a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c" exitCode=0 Nov 02 07:12:46.632855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.632825 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" Nov 02 07:12:46.633141 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.632846 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" event={"ID":"61149048-1e88-4b5f-ba34-07522f200893","Type":"ContainerDied","Data":"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c"} Nov 02 07:12:46.633141 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.632924 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4" event={"ID":"61149048-1e88-4b5f-ba34-07522f200893","Type":"ContainerDied","Data":"24dd363a318ea853298b66bbe56dee18479972a9663a4eccc165598bdff45da8"} Nov 02 07:12:46.633141 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.632947 6273 scope.go:122] "RemoveContainer" containerID="a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c" Nov 02 07:12:46.634300 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.634274 6273 generic.go:356] "Generic (PLEG): container finished" podID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerID="cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4" exitCode=0 Nov 02 07:12:46.634406 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.634306 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" event={"ID":"975e62b9-53b0-4b81-8814-c1abb06b4d87","Type":"ContainerDied","Data":"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4"} Nov 02 07:12:46.634406 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.634336 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" event={"ID":"975e62b9-53b0-4b81-8814-c1abb06b4d87","Type":"ContainerDied","Data":"1e2d44236acaf138862ae7b6b01b49a7f1d6162094fc7aefb5ffd0dcf21cd662"} Nov 02 07:12:46.634406 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.634359 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv" Nov 02 07:12:46.642499 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.642482 6273 scope.go:122] "RemoveContainer" containerID="a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c" Nov 02 07:12:46.642752 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:46.642730 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c\": container with ID starting with a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c not found: ID does not exist" containerID="a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c" Nov 02 07:12:46.642816 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.642759 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c"} err="failed to get container status \"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c\": rpc error: code = NotFound desc = could not find container \"a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c\": container with ID starting with a239ffc4c39f44ae7c3de8107eac6c55595273574c5d0cb826cbdc44b3a2332c not found: ID does not exist" Nov 02 07:12:46.642816 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.642775 6273 scope.go:122] "RemoveContainer" containerID="cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4" Nov 02 07:12:46.651031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.651010 6273 scope.go:122] "RemoveContainer" containerID="cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4" Nov 02 07:12:46.651297 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:12:46.651278 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4\": container with ID starting with cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4 not found: ID does not exist" containerID="cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4" Nov 02 07:12:46.651359 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.651304 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4"} err="failed to get container status \"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4\": rpc error: code = NotFound desc = could not find container \"cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4\": container with ID starting with cccb4c88ca12c1a5f9875b8090b9555c2c9cbe3502512015ec2ee697f7d7a3f4 not found: ID does not exist" Nov 02 07:12:46.786855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.786821 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:12:46.791119 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.791093 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-614951-86dd6d578qgdlv"] Nov 02 07:12:46.893368 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.893293 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:12:46.897303 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:46.897275 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-614951-5fd7cb9pg4"] Nov 02 07:12:48.177922 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:48.177892 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61149048-1e88-4b5f-ba34-07522f200893" path="/var/lib/kubelet/pods/61149048-1e88-4b5f-ba34-07522f200893/volumes" Nov 02 07:12:48.178314 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:48.178296 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" path="/var/lib/kubelet/pods/975e62b9-53b0-4b81-8814-c1abb06b4d87/volumes" Nov 02 07:12:53.194137 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:53.194109 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:12:54.043460 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:54.043425 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq"] Nov 02 07:12:54.094013 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:54.093728 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skdbq"] Nov 02 07:12:54.224432 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:12:54.224397 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20de29b6-4912-4732-9ce0-a4c76f63b881" path="/var/lib/kubelet/pods/20de29b6-4912-4732-9ce0-a4c76f63b881/volumes" Nov 02 07:13:03.438898 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:03.438821 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:13:08.552759 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:13:08.552721 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:13:15.082081 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:15.082029 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr"] Nov 02 07:13:15.127336 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:15.127308 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wmzxr"] Nov 02 07:13:16.178565 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:16.178530 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b175837-10c4-4441-8507-582ffa984b14" path="/var/lib/kubelet/pods/5b175837-10c4-4441-8507-582ffa984b14/volumes" Nov 02 07:13:27.647182 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:27.647153 6273 scope.go:122] "RemoveContainer" containerID="60f862aad4ea54f53d00ce8561fe89be076cb7e354694347c64d670773540814" Nov 02 07:13:27.655886 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:13:27.655866 6273 scope.go:122] "RemoveContainer" containerID="061ac7c338376c650f1966c8bcb09aa38804a4dde8cb5c8d6c1f370127893988" Nov 02 07:13:38.551432 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:13:38.551393 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:14:08.551466 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:14:08.551425 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:14:27.696955 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:27.696922 6273 scope.go:122] "RemoveContainer" containerID="8f25619c834db959b0aa841f1e1255df5d012176905564d4a70f686339653f18" Nov 02 07:14:27.705754 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:27.705734 6273 scope.go:122] "RemoveContainer" containerID="555aa0654eba454667561f5baa434009f76054c71451f7a5af63f702c5f733a2" Nov 02 07:14:27.713538 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:27.713521 6273 scope.go:122] "RemoveContainer" containerID="7caaab869d50da8628a606c4ab0ed69974d26b0dab3068c5612839c97bf73455" Nov 02 07:14:37.791444 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:37.791362 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:14:38.554175 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:14:38.554132 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:14:42.252612 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.252582 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x"] Nov 02 07:14:42.253235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253213 6273 cpu_manager.go:414] "containerMap: removing container" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" Nov 02 07:14:42.253287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253239 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" Nov 02 07:14:42.253287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253277 6273 cpu_manager.go:414] "containerMap: removing container" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" Nov 02 07:14:42.253354 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253285 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" Nov 02 07:14:42.253354 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253334 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7c065deb-6cff-42c4-aff0-597ed8a6d5d3" containerName="revoke-keys" Nov 02 07:14:42.253354 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253342 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c065deb-6cff-42c4-aff0-597ed8a6d5d3" containerName="revoke-keys" Nov 02 07:14:42.253451 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253428 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="975e62b9-53b0-4b81-8814-c1abb06b4d87" containerName="payload-processing" Nov 02 07:14:42.253451 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253446 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c065deb-6cff-42c4-aff0-597ed8a6d5d3" containerName="revoke-keys" Nov 02 07:14:42.253517 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.253456 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="61149048-1e88-4b5f-ba34-07522f200893" containerName="payload-pre-processing" Nov 02 07:14:42.258492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.258475 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.311948 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.311917 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x"] Nov 02 07:14:42.356665 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.356639 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:14:42.356828 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.356697 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:14:42.366181 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.366139 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:14:42.366296 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.366261 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:14:42.402107 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.402080 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.402238 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.402124 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.402238 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.402190 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5qb\" (UniqueName: \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.502924 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.502855 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.502924 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.502898 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.503138 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.502933 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5qb\" (UniqueName: \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.503666 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.503638 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.505139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.505119 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.557007 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.556961 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5qb\" (UniqueName: \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\") pod \"maas-api-revoke-sub-d576e68b-9fj2x\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.570782 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.570760 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:42.938151 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:42.938118 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x"] Nov 02 07:14:42.941415 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:14:42.941385 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed2087a_cea0_453f_82cb_eb7853dc9872.slice/crio-e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0 WatchSource:0}: Error finding container e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0: Status 404 returned error can't find the container with id e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0 Nov 02 07:14:43.044146 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:43.044116 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm"] Nov 02 07:14:43.046555 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:43.046534 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wwrwm"] Nov 02 07:14:43.257765 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:43.257726 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" event={"ID":"aed2087a-cea0-453f-82cb-eb7853dc9872","Type":"ContainerStarted","Data":"e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0"} Nov 02 07:14:44.178870 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:44.178838 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ff7ac6-ab05-4061-9f5b-552087ed605e" path="/var/lib/kubelet/pods/37ff7ac6-ab05-4061-9f5b-552087ed605e/volumes" Nov 02 07:14:44.264800 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:44.264771 6273 generic.go:356] "Generic (PLEG): container finished" podID="aed2087a-cea0-453f-82cb-eb7853dc9872" containerID="ade40f531a536a8b34832ac7a08f180cb0a7a71b7db1b1a2e2198007b9e5bfae" exitCode=0 Nov 02 07:14:44.265213 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:44.264863 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" event={"ID":"aed2087a-cea0-453f-82cb-eb7853dc9872","Type":"ContainerDied","Data":"ade40f531a536a8b34832ac7a08f180cb0a7a71b7db1b1a2e2198007b9e5bfae"} Nov 02 07:14:45.410275 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.410253 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:45.536867 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.536785 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\") pod \"aed2087a-cea0-453f-82cb-eb7853dc9872\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " Nov 02 07:14:45.536867 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.536839 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\" (UniqueName: \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\") pod \"aed2087a-cea0-453f-82cb-eb7853dc9872\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " Nov 02 07:14:45.537068 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.536906 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\") pod \"aed2087a-cea0-453f-82cb-eb7853dc9872\" (UID: \"aed2087a-cea0-453f-82cb-eb7853dc9872\") " Nov 02 07:14:45.537192 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.537170 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca" pod "aed2087a-cea0-453f-82cb-eb7853dc9872" (UID: "aed2087a-cea0-453f-82cb-eb7853dc9872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:14:45.537328 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.537309 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:14:45.539162 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.539129 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca" pod "aed2087a-cea0-453f-82cb-eb7853dc9872" (UID: "aed2087a-cea0-453f-82cb-eb7853dc9872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:14:45.539281 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.539217 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb" pod "aed2087a-cea0-453f-82cb-eb7853dc9872" (UID: "aed2087a-cea0-453f-82cb-eb7853dc9872"). InnerVolumeSpecName "kube-api-access-5w5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:14:45.638080 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.638054 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5qb\" (UniqueName: \"kubernetes.io/projected/aed2087a-cea0-453f-82cb-eb7853dc9872-kube-api-access-5w5qb\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:14:45.638080 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:45.638076 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed2087a-cea0-453f-82cb-eb7853dc9872-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:14:46.276235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:46.276207 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" Nov 02 07:14:46.276421 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:46.276257 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x" event={"ID":"aed2087a-cea0-453f-82cb-eb7853dc9872","Type":"ContainerDied","Data":"e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0"} Nov 02 07:14:46.276421 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:46.276306 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1306c62e245cb2db4401b8451a946bfcc3676006c8766f5815e436660c1b3a0" Nov 02 07:14:54.046095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:54.046059 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t"] Nov 02 07:14:54.092253 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:54.092206 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzk7t"] Nov 02 07:14:54.178198 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:54.178165 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be54940-9886-4aac-aa48-84f6756a658e" path="/var/lib/kubelet/pods/2be54940-9886-4aac-aa48-84f6756a658e/volumes" Nov 02 07:14:55.066170 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:14:55.066134 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:15:00.166198 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.166167 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:00.166739 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.166620 6273 cpu_manager.go:414] "containerMap: removing container" podUID="aed2087a-cea0-453f-82cb-eb7853dc9872" containerName="revoke-keys" Nov 02 07:15:00.166739 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.166633 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed2087a-cea0-453f-82cb-eb7853dc9872" containerName="revoke-keys" Nov 02 07:15:00.166857 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.166764 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed2087a-cea0-453f-82cb-eb7853dc9872" containerName="revoke-keys" Nov 02 07:15:00.170433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.170408 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.223744 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.223708 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:00.267451 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.267385 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:15:00.267628 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.267474 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:15:00.267628 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.267521 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:15:00.275859 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.275831 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gdk\" (UniqueName: \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.276236 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.276211 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.377231 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.377195 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gdk\" (UniqueName: \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.377414 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.377340 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.378193 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.378169 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.431282 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.431216 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gdk\" (UniqueName: \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\") pod \"maas-api-key-cleanup-29893395-j9rnj\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.486960 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.486934 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:00.860346 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:00.860246 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:00.863873 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:15:00.863834 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054fb164_a1bb_423e_9c85_73b85b2a13eb.slice/crio-39ea812f3f694cb7ced992160987e1dc7165cccbb6c51c8edebdf3699ad14aa5 WatchSource:0}: Error finding container 39ea812f3f694cb7ced992160987e1dc7165cccbb6c51c8edebdf3699ad14aa5: Status 404 returned error can't find the container with id 39ea812f3f694cb7ced992160987e1dc7165cccbb6c51c8edebdf3699ad14aa5 Nov 02 07:15:01.382362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:01.382324 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerStarted","Data":"ba135d994ea39bb2783aa57013855343ceaee8ae74600f342e08d57652639da6"} Nov 02 07:15:01.382967 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:01.382939 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerStarted","Data":"39ea812f3f694cb7ced992160987e1dc7165cccbb6c51c8edebdf3699ad14aa5"} Nov 02 07:15:01.491061 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:01.490390 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" podStartSLOduration=1.490371787 podStartE2EDuration="1.490371787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:01.48989454 +0000 UTC m=+3455.463229249" watchObservedRunningTime="2026-11-02 07:15:01.490371787 +0000 UTC m=+3455.463706472" Nov 02 07:15:02.411012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:02.410478 6273 generic.go:356] "Generic (PLEG): container finished" podID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerID="ba135d994ea39bb2783aa57013855343ceaee8ae74600f342e08d57652639da6" exitCode=60 Nov 02 07:15:02.411012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:02.410576 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerDied","Data":"ba135d994ea39bb2783aa57013855343ceaee8ae74600f342e08d57652639da6"} Nov 02 07:15:02.411012 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:02.410966 6273 scope.go:122] "RemoveContainer" containerID="ba135d994ea39bb2783aa57013855343ceaee8ae74600f342e08d57652639da6" Nov 02 07:15:03.416583 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:03.416550 6273 generic.go:356] "Generic (PLEG): container finished" podID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerID="2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac" exitCode=60 Nov 02 07:15:03.416983 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:03.416634 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerDied","Data":"2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac"} Nov 02 07:15:03.416983 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:03.416718 6273 scope.go:122] "RemoveContainer" containerID="ba135d994ea39bb2783aa57013855343ceaee8ae74600f342e08d57652639da6" Nov 02 07:15:03.416983 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:03.416865 6273 scope.go:122] "RemoveContainer" containerID="2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac" Nov 02 07:15:03.417154 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:15:03.417131 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893395-j9rnj_odh-ai-gateway-infra(054fb164-a1bb-423e-9c85-73b85b2a13eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" Nov 02 07:15:04.423805 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:04.423781 6273 scope.go:122] "RemoveContainer" containerID="2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac" Nov 02 07:15:04.424193 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:15:04.423980 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893395-j9rnj_odh-ai-gateway-infra(054fb164-a1bb-423e-9c85-73b85b2a13eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" Nov 02 07:15:04.643499 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:04.643469 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:15:08.551904 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:15:08.551861 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:15:14.172587 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:14.172555 6273 scope.go:122] "RemoveContainer" containerID="2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac" Nov 02 07:15:14.474953 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:14.474924 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerStarted","Data":"41f30878adf94016b22029d2758a2270ab1304958bf2305efb7a6455f4035a45"} Nov 02 07:15:15.481272 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:15.481237 6273 generic.go:356] "Generic (PLEG): container finished" podID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerID="41f30878adf94016b22029d2758a2270ab1304958bf2305efb7a6455f4035a45" exitCode=60 Nov 02 07:15:15.481689 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:15.481311 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerDied","Data":"41f30878adf94016b22029d2758a2270ab1304958bf2305efb7a6455f4035a45"} Nov 02 07:15:15.481689 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:15.481387 6273 scope.go:122] "RemoveContainer" containerID="2733d8b20baf2596d1ade2447dde9069849ad78c61a81df553a31118114497ac" Nov 02 07:15:15.481689 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:15.481648 6273 scope.go:122] "RemoveContainer" containerID="41f30878adf94016b22029d2758a2270ab1304958bf2305efb7a6455f4035a45" Nov 02 07:15:15.481895 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:15:15.481871 6273 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893395-j9rnj_odh-ai-gateway-infra(054fb164-a1bb-423e-9c85-73b85b2a13eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" Nov 02 07:15:15.600115 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:15.600087 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:16.622641 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.622619 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:16.766525 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.766448 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\") pod \"054fb164-a1bb-423e-9c85-73b85b2a13eb\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " Nov 02 07:15:16.766709 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.766611 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\" (UniqueName: \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\") pod \"054fb164-a1bb-423e-9c85-73b85b2a13eb\" (UID: \"054fb164-a1bb-423e-9c85-73b85b2a13eb\") " Nov 02 07:15:16.766855 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.766832 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca" pod "054fb164-a1bb-423e-9c85-73b85b2a13eb" (UID: "054fb164-a1bb-423e-9c85-73b85b2a13eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:15:16.768706 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.768684 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk" pod "054fb164-a1bb-423e-9c85-73b85b2a13eb" (UID: "054fb164-a1bb-423e-9c85-73b85b2a13eb"). InnerVolumeSpecName "kube-api-access-z2gdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:15:16.867562 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.867531 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2gdk\" (UniqueName: \"kubernetes.io/projected/054fb164-a1bb-423e-9c85-73b85b2a13eb-kube-api-access-z2gdk\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:15:16.867562 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:16.867557 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/054fb164-a1bb-423e-9c85-73b85b2a13eb-maas-api-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:15:17.493027 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:17.492979 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" event={"ID":"054fb164-a1bb-423e-9c85-73b85b2a13eb","Type":"ContainerDied","Data":"39ea812f3f694cb7ced992160987e1dc7165cccbb6c51c8edebdf3699ad14aa5"} Nov 02 07:15:17.493220 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:17.493036 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj" Nov 02 07:15:17.493220 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:17.493050 6273 scope.go:122] "RemoveContainer" containerID="41f30878adf94016b22029d2758a2270ab1304958bf2305efb7a6455f4035a45" Nov 02 07:15:17.646948 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:17.646914 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:17.693151 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:17.693121 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893395-j9rnj"] Nov 02 07:15:18.178568 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:18.178533 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" path="/var/lib/kubelet/pods/054fb164-a1bb-423e-9c85-73b85b2a13eb/volumes" Nov 02 07:15:24.051328 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:24.051297 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:15:27.739097 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:27.739067 6273 scope.go:122] "RemoveContainer" containerID="e2a3fdc56c614504363b3d694daf9ec0311a728750a777a02c90272c680d55a5" Nov 02 07:15:32.401670 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.401635 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:15:32.402126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402111 6273 cpu_manager.go:414] "containerMap: removing container" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402175 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402127 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402175 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402140 6273 cpu_manager.go:414] "containerMap: removing container" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402175 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402145 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402175 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402154 6273 cpu_manager.go:414] "containerMap: removing container" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402175 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402159 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402244 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402253 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.402338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.402259 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="054fb164-a1bb-423e-9c85-73b85b2a13eb" containerName="cleanup" Nov 02 07:15:32.406545 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.406524 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.504468 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.504434 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-75b25b\"" Nov 02 07:15:32.504640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.504472 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b46cca62-dockercfg-zdh72\"" Nov 02 07:15:32.505878 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.505856 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:15:32.515591 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.515565 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zwc\" (UniqueName: \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.515732 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.515619 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.557305 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.557278 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:15:32.560895 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.560879 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.616429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.616401 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.616768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.616574 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zwc\" (UniqueName: \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.617077 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.617059 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.623498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.623475 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:15:32.673039 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.672968 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zwc\" (UniqueName: \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\") pod \"payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.718096 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.718069 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.718221 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.718107 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxq4\" (UniqueName: \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.719887 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.719869 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:32.818743 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.818711 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.818884 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.818749 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxq4\" (UniqueName: \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.819405 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.819386 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.872497 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.872468 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxq4\" (UniqueName: \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\") pod \"payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:32.875132 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:32.875108 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:33.084561 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.084525 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:15:33.088167 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:15:33.088140 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb960ffec_3ffc_4fb8_b1cc_cf448557eaac.slice/crio-db32e1167da5b6db9aae43e34786eb84ca59f08f3de96e64571cc019f1e39d6c WatchSource:0}: Error finding container db32e1167da5b6db9aae43e34786eb84ca59f08f3de96e64571cc019f1e39d6c: Status 404 returned error can't find the container with id db32e1167da5b6db9aae43e34786eb84ca59f08f3de96e64571cc019f1e39d6c Nov 02 07:15:33.242278 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.242183 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:15:33.245021 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:15:33.244981 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0272d8_29a7_4524_af6c_8ae853af7900.slice/crio-2fb6a1e7d99041ac79ed9d8d375a834309d887559950cbbd758f13731edf858b WatchSource:0}: Error finding container 2fb6a1e7d99041ac79ed9d8d375a834309d887559950cbbd758f13731edf858b: Status 404 returned error can't find the container with id 2fb6a1e7d99041ac79ed9d8d375a834309d887559950cbbd758f13731edf858b Nov 02 07:15:33.582221 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.582117 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" event={"ID":"b960ffec-3ffc-4fb8-b1cc-cf448557eaac","Type":"ContainerStarted","Data":"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647"} Nov 02 07:15:33.582629 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.582266 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:33.582629 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.582311 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" event={"ID":"b960ffec-3ffc-4fb8-b1cc-cf448557eaac","Type":"ContainerStarted","Data":"db32e1167da5b6db9aae43e34786eb84ca59f08f3de96e64571cc019f1e39d6c"} Nov 02 07:15:33.583697 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.583679 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" event={"ID":"cd0272d8-29a7-4524-af6c-8ae853af7900","Type":"ContainerStarted","Data":"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea"} Nov 02 07:15:33.583789 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.583714 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" event={"ID":"cd0272d8-29a7-4524-af6c-8ae853af7900","Type":"ContainerStarted","Data":"2fb6a1e7d99041ac79ed9d8d375a834309d887559950cbbd758f13731edf858b"} Nov 02 07:15:33.583789 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.583776 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:33.689944 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.689894 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" podStartSLOduration=1.6898801890000001 podStartE2EDuration="1.689880189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:33.689050713 +0000 UTC m=+3487.662385405" watchObservedRunningTime="2026-11-02 07:15:33.689880189 +0000 UTC m=+3487.663214866" Nov 02 07:15:33.798592 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:33.798539 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" podStartSLOduration=1.798462682 podStartE2EDuration="1.798462682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:33.797773954 +0000 UTC m=+3487.771108649" watchObservedRunningTime="2026-11-02 07:15:33.798462682 +0000 UTC m=+3487.771797387" Nov 02 07:15:34.495431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.495394 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:15:34.499377 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.499353 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.550332 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.550299 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:15:34.550332 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.550307 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-75b25b-serving-cert\"" Nov 02 07:15:34.550483 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.550356 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:15:34.550483 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.550397 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 07:15:34.550483 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.550465 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 07:15:34.597812 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.597781 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:15:34.637974 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.637938 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfsh9\" (UniqueName: \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.638158 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.638116 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.638253 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.638233 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.638320 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.638288 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.739306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.739267 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.739528 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.739371 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsh9\" (UniqueName: \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.739528 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.739454 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.739620 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.739555 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.740833 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.740808 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.742675 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.742645 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.743112 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.743088 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.793724 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.793658 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsh9\" (UniqueName: \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\") pod \"maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:34.812669 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:34.812647 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:35.178376 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:35.178347 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:15:35.181276 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:15:35.181242 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8969e146_06b5_48b4_81d1_d7f383619e47.slice/crio-011e093464db9e0fd506b26c47a9fad7554da4c40624b03e8cf25be2b36e8589 WatchSource:0}: Error finding container 011e093464db9e0fd506b26c47a9fad7554da4c40624b03e8cf25be2b36e8589: Status 404 returned error can't find the container with id 011e093464db9e0fd506b26c47a9fad7554da4c40624b03e8cf25be2b36e8589 Nov 02 07:15:35.595202 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:35.595116 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" event={"ID":"8969e146-06b5-48b4-81d1-d7f383619e47","Type":"ContainerStarted","Data":"011e093464db9e0fd506b26c47a9fad7554da4c40624b03e8cf25be2b36e8589"} Nov 02 07:15:36.601347 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:36.601256 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" event={"ID":"8969e146-06b5-48b4-81d1-d7f383619e47","Type":"ContainerStarted","Data":"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d"} Nov 02 07:15:36.601347 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:36.601323 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:36.730843 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:36.730781 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" podStartSLOduration=1.671531092 podStartE2EDuration="2.730765319s" totalImagesPullingTime="1.059234227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:36.730192797 +0000 UTC m=+3490.703527481" watchObservedRunningTime="2026-11-02 07:15:36.730765319 +0000 UTC m=+3490.704100004" Nov 02 07:15:38.552276 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:15:38.552227 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:15:42.578166 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:42.578126 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:15:42.612699 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:42.612666 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:15:44.591814 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:44.591781 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:15:44.592244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:44.591846 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:15:57.801899 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.801865 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:15:57.805436 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.805417 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855114 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855086 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-75b25b\"/\"kube-root-ca.crt\"" Nov 02 07:15:57.855365 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855228 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-75b25b\"/\"openshift-service-ca.crt\"" Nov 02 07:15:57.855365 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855243 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-75b25b\"/\"default-dockercfg-rtfp7\"" Nov 02 07:15:57.855365 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855317 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-75b25b\"/\"simulator-single-main-75b25b-kserve-self-signed-certs\"" Nov 02 07:15:57.855488 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855373 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855488 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855403 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855488 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855452 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855582 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855547 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855673 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855654 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.855709 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.855692 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrhj\" (UniqueName: \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.860522 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.860499 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:15:57.956576 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956533 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrhj\" (UniqueName: \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.956872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956671 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.956872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956704 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.956872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956764 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.956872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956788 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.956872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.956846 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.957244 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.957217 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.957330 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.957251 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.957330 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.957225 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.959339 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.959314 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:57.972730 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:57.972707 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:58.010326 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.010303 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrhj\" (UniqueName: \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\") pod \"simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:58.118026 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.117940 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:15:58.486948 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.486918 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:15:58.489334 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:15:58.489302 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8552522d_0419_4781_96bc_bb31f2e74a22.slice/crio-9f7c2f978ce8ce33b8c0bc3d580e2c0733cc2ec3d2e39caffa6e30c7e83dc391 WatchSource:0}: Error finding container 9f7c2f978ce8ce33b8c0bc3d580e2c0733cc2ec3d2e39caffa6e30c7e83dc391: Status 404 returned error can't find the container with id 9f7c2f978ce8ce33b8c0bc3d580e2c0733cc2ec3d2e39caffa6e30c7e83dc391 Nov 02 07:15:58.713458 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.713411 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" event={"ID":"8552522d-0419-4781-96bc-bb31f2e74a22","Type":"ContainerStarted","Data":"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d"} Nov 02 07:15:58.713619 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.713464 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" event={"ID":"8552522d-0419-4781-96bc-bb31f2e74a22","Type":"ContainerStarted","Data":"9f7c2f978ce8ce33b8c0bc3d580e2c0733cc2ec3d2e39caffa6e30c7e83dc391"} Nov 02 07:15:58.822355 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:58.822259 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" podStartSLOduration=1.8222450289999998 podStartE2EDuration="1.822245029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:58.820147875 +0000 UTC m=+3512.793482558" watchObservedRunningTime="2026-11-02 07:15:58.822245029 +0000 UTC m=+3512.795579702" Nov 02 07:15:59.118414 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:15:59.118325 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:16:02.674682 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:02.674647 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:16:08.118433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:08.118344 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:16:08.130646 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:08.130626 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:16:08.551801 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:16:08.551764 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:16:08.774391 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:08.774358 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:16:27.763285 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:27.763255 6273 scope.go:122] "RemoveContainer" containerID="c0e1d31da45460ae243704f0525075c208e7a9a1a787c885fd3db246359c56bb" Nov 02 07:16:27.773932 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:27.773907 6273 scope.go:122] "RemoveContainer" containerID="68989d93c814033cfd19cbb62d96ac15d747cbcf57956ae6c3e31d7f14c5a3cd" Nov 02 07:16:27.781651 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:27.781633 6273 scope.go:122] "RemoveContainer" containerID="deefbaadcbbc30f245a7bb640f81dc37aee75d50cb4c3eb38544df7ce60f46eb" Nov 02 07:16:29.579511 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:29.579478 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:16:38.551344 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:16:38.551307 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:16:55.440585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:55.440545 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:16:58.406147 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:16:58.406114 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:17:04.943928 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:04.943891 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:17:04.944510 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:04.944132 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" podUID="8969e146-06b5-48b4-81d1-d7f383619e47" containerName="maas-api" containerID="cri-o://e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d" gracePeriod=30 Nov 02 07:17:05.340079 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:05.339999 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:17:05.412027 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:05.411975 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:17:06.207178 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.207154 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:17:06.287176 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287144 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\") pod \"8969e146-06b5-48b4-81d1-d7f383619e47\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " Nov 02 07:17:06.287317 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287186 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\") pod \"8969e146-06b5-48b4-81d1-d7f383619e47\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " Nov 02 07:17:06.287393 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287338 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\" (UniqueName: \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\") pod \"8969e146-06b5-48b4-81d1-d7f383619e47\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " Nov 02 07:17:06.287455 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287402 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\") pod \"8969e146-06b5-48b4-81d1-d7f383619e47\" (UID: \"8969e146-06b5-48b4-81d1-d7f383619e47\") " Nov 02 07:17:06.287562 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287532 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca" pod "8969e146-06b5-48b4-81d1-d7f383619e47" (UID: "8969e146-06b5-48b4-81d1-d7f383619e47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:17:06.287702 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.287692 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8969e146-06b5-48b4-81d1-d7f383619e47-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:06.289287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.289257 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs" pod "8969e146-06b5-48b4-81d1-d7f383619e47" (UID: "8969e146-06b5-48b4-81d1-d7f383619e47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:17:06.289388 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.289367 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9" pod "8969e146-06b5-48b4-81d1-d7f383619e47" (UID: "8969e146-06b5-48b4-81d1-d7f383619e47"). InnerVolumeSpecName "kube-api-access-tfsh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:06.289431 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.289388 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls" pod "8969e146-06b5-48b4-81d1-d7f383619e47" (UID: "8969e146-06b5-48b4-81d1-d7f383619e47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:17:06.388472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.388441 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfsh9\" (UniqueName: \"kubernetes.io/projected/8969e146-06b5-48b4-81d1-d7f383619e47-kube-api-access-tfsh9\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:06.388472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.388468 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:06.388472 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:06.388478 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8969e146-06b5-48b4-81d1-d7f383619e47-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:07.056079 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.056047 6273 generic.go:356] "Generic (PLEG): container finished" podID="8969e146-06b5-48b4-81d1-d7f383619e47" containerID="e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d" exitCode=0 Nov 02 07:17:07.056259 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.056092 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" Nov 02 07:17:07.056259 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.056133 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" event={"ID":"8969e146-06b5-48b4-81d1-d7f383619e47","Type":"ContainerDied","Data":"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d"} Nov 02 07:17:07.056259 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.056191 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn" event={"ID":"8969e146-06b5-48b4-81d1-d7f383619e47","Type":"ContainerDied","Data":"011e093464db9e0fd506b26c47a9fad7554da4c40624b03e8cf25be2b36e8589"} Nov 02 07:17:07.056259 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.056219 6273 scope.go:122] "RemoveContainer" containerID="e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d" Nov 02 07:17:07.066164 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.065935 6273 scope.go:122] "RemoveContainer" containerID="e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d" Nov 02 07:17:07.066290 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:07.066271 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d\": container with ID starting with e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d not found: ID does not exist" containerID="e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d" Nov 02 07:17:07.066331 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.066298 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d"} err="failed to get container status \"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d\": rpc error: code = NotFound desc = could not find container \"e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d\": container with ID starting with e7e4df81153e281cb0ab18d2fa41a7088038651767c2d6d023caaa9461c4283d not found: ID does not exist" Nov 02 07:17:07.209672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.209643 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:17:07.256054 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:07.256030 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-75b25b-569d95449-bwgzn"] Nov 02 07:17:08.178366 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:08.178333 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8969e146-06b5-48b4-81d1-d7f383619e47" path="/var/lib/kubelet/pods/8969e146-06b5-48b4-81d1-d7f383619e47/volumes" Nov 02 07:17:08.551849 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:08.551775 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:17:09.745867 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:09.745821 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:17:10.341095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:10.341060 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" containerID="cri-o://7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea" gracePeriod=25 Nov 02 07:17:10.412723 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:10.412669 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" containerID="cri-o://d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647" gracePeriod=25 Nov 02 07:17:13.426654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.426606 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" podUID="8552522d-0419-4781-96bc-bb31f2e74a22" containerName="main" containerID="cri-o://f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d" gracePeriod=45 Nov 02 07:17:13.682976 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.682917 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:17:13.761576 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761547 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761612 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761636 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\" (UniqueName: \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761681 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\" (UniqueName: \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761701 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761810 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\") pod \"8552522d-0419-4781-96bc-bb31f2e74a22\" (UID: \"8552522d-0419-4781-96bc-bb31f2e74a22\") " Nov 02 07:17:13.761940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761810 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:13.761940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.761865 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:13.762139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.762076 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:13.762272 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.762252 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:13.762272 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.762270 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:13.762466 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.762280 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:13.763949 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.763927 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:13.764066 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.764005 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:17:13.764112 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.764095 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj" pod "8552522d-0419-4781-96bc-bb31f2e74a22" (UID: "8552522d-0419-4781-96bc-bb31f2e74a22"). InnerVolumeSpecName "kube-api-access-jlrhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:13.863671 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.863633 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8552522d-0419-4781-96bc-bb31f2e74a22-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:13.863671 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.863665 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlrhj\" (UniqueName: \"kubernetes.io/projected/8552522d-0419-4781-96bc-bb31f2e74a22-kube-api-access-jlrhj\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:13.863671 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:13.863675 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8552522d-0419-4781-96bc-bb31f2e74a22-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:14.091113 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.091024 6273 generic.go:356] "Generic (PLEG): container finished" podID="8552522d-0419-4781-96bc-bb31f2e74a22" containerID="f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d" exitCode=0 Nov 02 07:17:14.091113 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.091069 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" Nov 02 07:17:14.091113 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.091078 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" event={"ID":"8552522d-0419-4781-96bc-bb31f2e74a22","Type":"ContainerDied","Data":"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d"} Nov 02 07:17:14.091113 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.091108 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9" event={"ID":"8552522d-0419-4781-96bc-bb31f2e74a22","Type":"ContainerDied","Data":"9f7c2f978ce8ce33b8c0bc3d580e2c0733cc2ec3d2e39caffa6e30c7e83dc391"} Nov 02 07:17:14.091433 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.091125 6273 scope.go:122] "RemoveContainer" containerID="f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d" Nov 02 07:17:14.100496 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.100475 6273 scope.go:122] "RemoveContainer" containerID="f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d" Nov 02 07:17:14.100794 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:14.100765 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d\": container with ID starting with f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d not found: ID does not exist" containerID="f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d" Nov 02 07:17:14.100852 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.100804 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d"} err="failed to get container status \"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d\": rpc error: code = NotFound desc = could not find container \"f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d\": container with ID starting with f6b5465d2535bb019a191e1b8f4ab7fe7160de615518a5b529b8b52913b4f59d not found: ID does not exist" Nov 02 07:17:14.243556 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.243528 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:17:14.247838 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.247816 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-75b25b/simulator-single-main-75b25b-kserve-db4bf69d9-fkgz9"] Nov 02 07:17:14.590138 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.590102 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:17:14.590530 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:14.590102 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:17:16.178490 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:16.178456 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8552522d-0419-4781-96bc-bb31f2e74a22" path="/var/lib/kubelet/pods/8552522d-0419-4781-96bc-bb31f2e74a22/volumes" Nov 02 07:17:23.401845 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.401809 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:17:23.402310 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402294 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8552522d-0419-4781-96bc-bb31f2e74a22" containerName="main" Nov 02 07:17:23.402357 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402311 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8552522d-0419-4781-96bc-bb31f2e74a22" containerName="main" Nov 02 07:17:23.402357 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402338 6273 cpu_manager.go:414] "containerMap: removing container" podUID="8969e146-06b5-48b4-81d1-d7f383619e47" containerName="maas-api" Nov 02 07:17:23.402357 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402343 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8969e146-06b5-48b4-81d1-d7f383619e47" containerName="maas-api" Nov 02 07:17:23.402461 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402405 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8552522d-0419-4781-96bc-bb31f2e74a22" containerName="main" Nov 02 07:17:23.402461 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.402418 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="8969e146-06b5-48b4-81d1-d7f383619e47" containerName="maas-api" Nov 02 07:17:23.406962 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.406942 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.503209 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.503174 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7d585e03-kserve-self-signed-certs\"" Nov 02 07:17:23.504663 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.504640 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:17:23.554404 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554371 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.554588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554409 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.554588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554506 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.554588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554525 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.554588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554543 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.554588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.554565 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77s9\" (UniqueName: \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.655972 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.655887 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.655972 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.655930 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656033 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656062 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656086 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656240 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656117 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77s9\" (UniqueName: \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656449 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656333 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656449 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656365 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.656449 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.656429 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.658171 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.658151 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.658421 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.658404 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.709363 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.709326 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77s9\" (UniqueName: \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\") pod \"e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:23.719031 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:23.719006 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:24.106137 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:24.106115 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:17:24.108840 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:17:24.108806 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3cf4834_7618_45fe_9c12_2ce558a50d89.slice/crio-637ebb83339a87ebb9a545062d5136e0eb3c14a1fb57ea5b7ff759f0aaa0f529 WatchSource:0}: Error finding container 637ebb83339a87ebb9a545062d5136e0eb3c14a1fb57ea5b7ff759f0aaa0f529: Status 404 returned error can't find the container with id 637ebb83339a87ebb9a545062d5136e0eb3c14a1fb57ea5b7ff759f0aaa0f529 Nov 02 07:17:24.143028 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:24.142975 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" event={"ID":"a3cf4834-7618-45fe-9c12-2ce558a50d89","Type":"ContainerStarted","Data":"637ebb83339a87ebb9a545062d5136e0eb3c14a1fb57ea5b7ff759f0aaa0f529"} Nov 02 07:17:24.590249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:24.590215 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:17:24.590701 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:24.590299 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:17:25.149935 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:25.149900 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" event={"ID":"a3cf4834-7618-45fe-9c12-2ce558a50d89","Type":"ContainerStarted","Data":"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0"} Nov 02 07:17:25.259322 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:25.259265 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" podStartSLOduration=2.259248777 podStartE2EDuration="2.259248777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:17:25.257199131 +0000 UTC m=+3599.230533816" watchObservedRunningTime="2026-11-02 07:17:25.259248777 +0000 UTC m=+3599.232583451" Nov 02 07:17:25.719524 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:25.719491 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:28.075352 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:28.075262 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:17:31.032160 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.031765 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:17:31.039204 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.039183 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:17:31.138508 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138392 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\") pod \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " Nov 02 07:17:31.138508 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138460 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\") pod \"cd0272d8-29a7-4524-af6c-8ae853af7900\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " Nov 02 07:17:31.138724 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138610 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\" (UniqueName: \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\") pod \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\" (UID: \"b960ffec-3ffc-4fb8-b1cc-cf448557eaac\") " Nov 02 07:17:31.138724 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138692 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\" (UniqueName: \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\") pod \"cd0272d8-29a7-4524-af6c-8ae853af7900\" (UID: \"cd0272d8-29a7-4524-af6c-8ae853af7900\") " Nov 02 07:17:31.139217 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138854 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume" pod "b960ffec-3ffc-4fb8-b1cc-cf448557eaac" (UID: "b960ffec-3ffc-4fb8-b1cc-cf448557eaac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:17:31.139217 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.138977 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume" pod "cd0272d8-29a7-4524-af6c-8ae853af7900" (UID: "cd0272d8-29a7-4524-af6c-8ae853af7900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:17:31.139217 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.139113 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:31.139217 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.139133 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0272d8-29a7-4524-af6c-8ae853af7900-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:31.141263 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.141236 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4" pod "cd0272d8-29a7-4524-af6c-8ae853af7900" (UID: "cd0272d8-29a7-4524-af6c-8ae853af7900"). InnerVolumeSpecName "kube-api-access-5wxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:31.141419 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.141396 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc" pod "b960ffec-3ffc-4fb8-b1cc-cf448557eaac" (UID: "b960ffec-3ffc-4fb8-b1cc-cf448557eaac"). InnerVolumeSpecName "kube-api-access-85zwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:31.216065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.215280 6273 generic.go:356] "Generic (PLEG): container finished" podID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerID="7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea" exitCode=0 Nov 02 07:17:31.216065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.215434 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" event={"ID":"cd0272d8-29a7-4524-af6c-8ae853af7900","Type":"ContainerDied","Data":"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea"} Nov 02 07:17:31.216065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.215484 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" event={"ID":"cd0272d8-29a7-4524-af6c-8ae853af7900","Type":"ContainerDied","Data":"2fb6a1e7d99041ac79ed9d8d375a834309d887559950cbbd758f13731edf858b"} Nov 02 07:17:31.216065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.215516 6273 scope.go:122] "RemoveContainer" containerID="7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea" Nov 02 07:17:31.216065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.215667 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4" Nov 02 07:17:31.217937 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.217914 6273 generic.go:356] "Generic (PLEG): container finished" podID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerID="d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647" exitCode=0 Nov 02 07:17:31.218129 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.218110 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" event={"ID":"b960ffec-3ffc-4fb8-b1cc-cf448557eaac","Type":"ContainerDied","Data":"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647"} Nov 02 07:17:31.218251 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.218155 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" event={"ID":"b960ffec-3ffc-4fb8-b1cc-cf448557eaac","Type":"ContainerDied","Data":"db32e1167da5b6db9aae43e34786eb84ca59f08f3de96e64571cc019f1e39d6c"} Nov 02 07:17:31.218251 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.218220 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n" Nov 02 07:17:31.228620 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.228575 6273 scope.go:122] "RemoveContainer" containerID="7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea" Nov 02 07:17:31.229660 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:31.228951 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea\": container with ID starting with 7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea not found: ID does not exist" containerID="7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea" Nov 02 07:17:31.229660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.229011 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea"} err="failed to get container status \"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea\": rpc error: code = NotFound desc = could not find container \"7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea\": container with ID starting with 7fbf8ec1e444a61bb6b4a93f6d3210a99b0edd75d0fff2ceffeb15d7d090b2ea not found: ID does not exist" Nov 02 07:17:31.229660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.229037 6273 scope.go:122] "RemoveContainer" containerID="d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647" Nov 02 07:17:31.241631 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.241602 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85zwc\" (UniqueName: \"kubernetes.io/projected/b960ffec-3ffc-4fb8-b1cc-cf448557eaac-kube-api-access-85zwc\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:31.241742 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.241636 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxq4\" (UniqueName: \"kubernetes.io/projected/cd0272d8-29a7-4524-af6c-8ae853af7900-kube-api-access-5wxq4\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:31.244769 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.244752 6273 scope.go:122] "RemoveContainer" containerID="d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647" Nov 02 07:17:31.245103 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:31.245081 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647\": container with ID starting with d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647 not found: ID does not exist" containerID="d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647" Nov 02 07:17:31.245173 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.245111 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647"} err="failed to get container status \"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647\": rpc error: code = NotFound desc = could not find container \"d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647\": container with ID starting with d7e68de08f58d951be296f8fcad7dc7cc0634ea7b7377f1287605cb42a89f647 not found: ID does not exist" Nov 02 07:17:31.371982 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.371947 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:17:31.374095 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.374060 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-75b25b-75c86598cqs4w4"] Nov 02 07:17:31.475915 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.475845 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:17:31.480733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:31.480707 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-75b25b-7c7f4tzj2n"] Nov 02 07:17:32.177662 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:32.177629 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" path="/var/lib/kubelet/pods/b960ffec-3ffc-4fb8-b1cc-cf448557eaac/volumes" Nov 02 07:17:32.178064 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:32.177983 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" path="/var/lib/kubelet/pods/cd0272d8-29a7-4524-af6c-8ae853af7900/volumes" Nov 02 07:17:33.719778 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:33.719700 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:33.732042 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:33.732022 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:34.243760 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:34.243734 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:17:38.552772 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:17:38.552739 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:17:42.043113 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:42.043076 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw"] Nov 02 07:17:42.089395 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:42.089366 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xlgpw"] Nov 02 07:17:42.178443 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:17:42.178407 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c065deb-6cff-42c4-aff0-597ed8a6d5d3" path="/var/lib/kubelet/pods/7c065deb-6cff-42c4-aff0-597ed8a6d5d3/volumes" Nov 02 07:18:03.115435 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115400 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:18:03.115819 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115791 6273 cpu_manager.go:414] "containerMap: removing container" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" Nov 02 07:18:03.115819 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115802 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" Nov 02 07:18:03.115940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115828 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" Nov 02 07:18:03.115940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115834 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" Nov 02 07:18:03.115940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115904 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0272d8-29a7-4524-af6c-8ae853af7900" containerName="payload-processing" Nov 02 07:18:03.115940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.115917 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="b960ffec-3ffc-4fb8-b1cc-cf448557eaac" containerName="payload-pre-processing" Nov 02 07:18:03.119146 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.119128 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.214329 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.214300 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7d585e03-kserve-self-signed-certs\"" Nov 02 07:18:03.216412 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.216381 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:18:03.233939 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.233913 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.234064 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.233961 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq8w\" (UniqueName: \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.234126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.234058 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.234126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.234098 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.234205 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.234174 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.234252 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.234200 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335515 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335479 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335515 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335517 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq8w\" (UniqueName: \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335538 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335557 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335610 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.335779 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.335626 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.336144 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.336113 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.336144 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.336132 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.336290 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.336179 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.337807 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.337787 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.338137 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.338121 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.388827 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.388762 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq8w\" (UniqueName: \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\") pod \"e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.431222 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.431197 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:03.801082 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:03.801047 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:18:03.804298 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:18:03.804270 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc315397d_390e_4515_add7_628d73fae500.slice/crio-bb9494213dabc99c90d5697214db6d7b2fc8e5f435b8b138ed6a9640a73cc35b WatchSource:0}: Error finding container bb9494213dabc99c90d5697214db6d7b2fc8e5f435b8b138ed6a9640a73cc35b: Status 404 returned error can't find the container with id bb9494213dabc99c90d5697214db6d7b2fc8e5f435b8b138ed6a9640a73cc35b Nov 02 07:18:04.384381 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:04.384340 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" event={"ID":"c315397d-390e-4515-add7-628d73fae500","Type":"ContainerStarted","Data":"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7"} Nov 02 07:18:04.384768 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:04.384394 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" event={"ID":"c315397d-390e-4515-add7-628d73fae500","Type":"ContainerStarted","Data":"bb9494213dabc99c90d5697214db6d7b2fc8e5f435b8b138ed6a9640a73cc35b"} Nov 02 07:18:04.431569 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:04.431537 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:04.491730 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:04.491667 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" podStartSLOduration=1.491650613 podStartE2EDuration="1.491650613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:18:04.490758304 +0000 UTC m=+3638.464092989" watchObservedRunningTime="2026-11-02 07:18:04.491650613 +0000 UTC m=+3638.464985298" Nov 02 07:18:07.964127 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:07.964091 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:18:08.553349 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:18:08.553301 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:18:13.431925 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:13.431884 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:13.444967 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:13.444938 6273 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:14.443256 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:14.443224 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:18:34.526348 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.526306 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh"] Nov 02 07:18:34.531749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.531725 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.543092 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.543068 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.543210 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.543113 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tng4r\" (UniqueName: \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.543291 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.543265 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.582623 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.582592 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:18:34.582732 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.582636 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:18:34.628897 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.628862 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:18:34.629094 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.628947 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:18:34.630510 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.630490 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh"] Nov 02 07:18:34.644599 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.644574 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.644718 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.644616 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tng4r\" (UniqueName: \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.644718 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.644664 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.645253 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.645224 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.647352 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.647330 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.699146 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.699123 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tng4r\" (UniqueName: \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\") pod \"maas-api-revoke-sub-452b6d7b-wkrsh\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:34.845237 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:34.845154 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:35.206332 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:35.206182 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh"] Nov 02 07:18:35.209359 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:18:35.209330 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b275d76_d355_4455_a112_cf44a7be3e3e.slice/crio-f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323 WatchSource:0}: Error finding container f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323: Status 404 returned error can't find the container with id f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323 Nov 02 07:18:35.548036 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:35.547864 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" event={"ID":"1b275d76-d355-4455-a112-cf44a7be3e3e","Type":"ContainerStarted","Data":"a37fe7a2667460366bf020d46e93c1f508b56a1da12ad7789651d89f48d119eb"} Nov 02 07:18:35.548036 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:35.547929 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" event={"ID":"1b275d76-d355-4455-a112-cf44a7be3e3e","Type":"ContainerStarted","Data":"f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323"} Nov 02 07:18:35.655022 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:35.654948 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" podStartSLOduration=1.654933287 podStartE2EDuration="1.654933287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:18:35.653396298 +0000 UTC m=+3669.626730982" watchObservedRunningTime="2026-11-02 07:18:35.654933287 +0000 UTC m=+3669.628267970" Nov 02 07:18:36.168813 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:36.168779 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:18:36.553999 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:36.553964 6273 generic.go:356] "Generic (PLEG): container finished" podID="1b275d76-d355-4455-a112-cf44a7be3e3e" containerID="a37fe7a2667460366bf020d46e93c1f508b56a1da12ad7789651d89f48d119eb" exitCode=0 Nov 02 07:18:36.554417 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:36.554035 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" event={"ID":"1b275d76-d355-4455-a112-cf44a7be3e3e","Type":"ContainerDied","Data":"a37fe7a2667460366bf020d46e93c1f508b56a1da12ad7789651d89f48d119eb"} Nov 02 07:18:37.760847 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.760787 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:37.777098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.776275 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\") pod \"1b275d76-d355-4455-a112-cf44a7be3e3e\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " Nov 02 07:18:37.777098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.776359 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\") pod \"1b275d76-d355-4455-a112-cf44a7be3e3e\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " Nov 02 07:18:37.777098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.776427 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\" (UniqueName: \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\") pod \"1b275d76-d355-4455-a112-cf44a7be3e3e\" (UID: \"1b275d76-d355-4455-a112-cf44a7be3e3e\") " Nov 02 07:18:37.778408 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.778285 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca" pod "1b275d76-d355-4455-a112-cf44a7be3e3e" (UID: "1b275d76-d355-4455-a112-cf44a7be3e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:18:37.779427 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.779390 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:18:37.816860 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.816549 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r" pod "1b275d76-d355-4455-a112-cf44a7be3e3e" (UID: "1b275d76-d355-4455-a112-cf44a7be3e3e"). InnerVolumeSpecName "kube-api-access-tng4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:18:37.817060 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.817022 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca" pod "1b275d76-d355-4455-a112-cf44a7be3e3e" (UID: "1b275d76-d355-4455-a112-cf44a7be3e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:18:37.880719 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.880638 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b275d76-d355-4455-a112-cf44a7be3e3e-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:18:37.880719 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:37.880681 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tng4r\" (UniqueName: \"kubernetes.io/projected/1b275d76-d355-4455-a112-cf44a7be3e3e-kube-api-access-tng4r\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:18:38.607105 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:38.606711 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" event={"ID":"1b275d76-d355-4455-a112-cf44a7be3e3e","Type":"ContainerDied","Data":"f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323"} Nov 02 07:18:38.607105 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:38.606774 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f366df4f41be7d9141a66dc122b80677ca1b6bbc8531dcc44bd8148b44efa323" Nov 02 07:18:38.607105 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:38.606842 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh" Nov 02 07:18:38.616676 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:18:38.616646 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:18:47.147030 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:47.146980 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:18:51.464534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:51.464502 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:18:58.822378 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:18:58.822345 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:19:02.167353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.167317 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerName="main" containerID="cri-o://f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0" gracePeriod=45 Nov 02 07:19:02.233285 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.233252 6273 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": dial tcp 10.134.0.184:8000: connect: connection refused" Nov 02 07:19:02.419411 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.419359 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:19:02.572401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572371 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572424 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572450 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572475 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572559 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572517 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\" (UniqueName: \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572798 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572562 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\") pod \"a3cf4834-7618-45fe-9c12-2ce558a50d89\" (UID: \"a3cf4834-7618-45fe-9c12-2ce558a50d89\") " Nov 02 07:19:02.572798 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572713 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:02.572798 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572724 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:02.572798 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572769 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:02.573019 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.572968 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.573019 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.573006 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.573019 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.573018 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.574628 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.574602 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:02.574742 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.574715 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:02.574811 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.574788 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9" pod "a3cf4834-7618-45fe-9c12-2ce558a50d89" (UID: "a3cf4834-7618-45fe-9c12-2ce558a50d89"). InnerVolumeSpecName "kube-api-access-r77s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:19:02.673796 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.673743 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77s9\" (UniqueName: \"kubernetes.io/projected/a3cf4834-7618-45fe-9c12-2ce558a50d89-kube-api-access-r77s9\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.673796 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.673764 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cf4834-7618-45fe-9c12-2ce558a50d89-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.673796 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.673774 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cf4834-7618-45fe-9c12-2ce558a50d89-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:02.745810 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.745778 6273 generic.go:356] "Generic (PLEG): container finished" podID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerID="f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0" exitCode=0 Nov 02 07:19:02.745912 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.745831 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" Nov 02 07:19:02.745912 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.745868 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" event={"ID":"a3cf4834-7618-45fe-9c12-2ce558a50d89","Type":"ContainerDied","Data":"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0"} Nov 02 07:19:02.745912 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.745907 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp" event={"ID":"a3cf4834-7618-45fe-9c12-2ce558a50d89","Type":"ContainerDied","Data":"637ebb83339a87ebb9a545062d5136e0eb3c14a1fb57ea5b7ff759f0aaa0f529"} Nov 02 07:19:02.746093 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.745926 6273 scope.go:122] "RemoveContainer" containerID="f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0" Nov 02 07:19:02.754895 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.754878 6273 scope.go:122] "RemoveContainer" containerID="f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0" Nov 02 07:19:02.755383 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:19:02.755359 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0\": container with ID starting with f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0 not found: ID does not exist" containerID="f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0" Nov 02 07:19:02.755469 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.755395 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0"} err="failed to get container status \"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0\": rpc error: code = NotFound desc = could not find container \"f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0\": container with ID starting with f1f22b4b3280bc5470cb0a32a0bb6060315d526e4ba216d07b6d91330e8570f0 not found: ID does not exist" Nov 02 07:19:02.902139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.901346 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:19:02.907009 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.905035 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7d585e03-kserve-68967f5cb5-dtsxp"] Nov 02 07:19:02.947560 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.947491 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm"] Nov 02 07:19:02.948034 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948017 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1b275d76-d355-4455-a112-cf44a7be3e3e" containerName="revoke-keys" Nov 02 07:19:02.948134 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948037 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b275d76-d355-4455-a112-cf44a7be3e3e" containerName="revoke-keys" Nov 02 07:19:02.948134 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948071 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerName="main" Nov 02 07:19:02.948134 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948079 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerName="main" Nov 02 07:19:02.948308 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948222 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b275d76-d355-4455-a112-cf44a7be3e3e" containerName="revoke-keys" Nov 02 07:19:02.948308 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.948241 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" containerName="main" Nov 02 07:19:02.952700 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:02.952683 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.004003 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.003962 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:19:03.004203 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.004159 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:19:03.004310 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.004291 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:19:03.004854 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.004371 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:19:03.006814 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.006748 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm"] Nov 02 07:19:03.077895 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.077855 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.078091 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.077910 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.078091 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.078060 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b8l\" (UniqueName: \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.179157 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.179130 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.179522 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.179173 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b8l\" (UniqueName: \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.179522 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.179250 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.179826 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.179807 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.181249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.181228 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.232630 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.232570 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b8l\" (UniqueName: \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\") pod \"maas-api-revoke-sub-3e31a506-z59gm\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.276115 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.276095 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:03.637370 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.637295 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm"] Nov 02 07:19:03.640776 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:19:03.640744 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1887eb19_efa5_429e_bb20_992e7144f6f6.slice/crio-853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5 WatchSource:0}: Error finding container 853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5: Status 404 returned error can't find the container with id 853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5 Nov 02 07:19:03.752891 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:03.752859 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" event={"ID":"1887eb19-efa5-429e-bb20-992e7144f6f6","Type":"ContainerStarted","Data":"853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5"} Nov 02 07:19:04.178173 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:04.178139 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cf4834-7618-45fe-9c12-2ce558a50d89" path="/var/lib/kubelet/pods/a3cf4834-7618-45fe-9c12-2ce558a50d89/volumes" Nov 02 07:19:04.532864 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:04.532835 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:19:04.760172 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:04.760091 6273 generic.go:356] "Generic (PLEG): container finished" podID="1887eb19-efa5-429e-bb20-992e7144f6f6" containerID="8cfcf85a2c8c424691083db530a0ef3589f6849f2e56dbdb1674badf6295e275" exitCode=0 Nov 02 07:19:04.760172 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:04.760125 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" event={"ID":"1887eb19-efa5-429e-bb20-992e7144f6f6","Type":"ContainerDied","Data":"8cfcf85a2c8c424691083db530a0ef3589f6849f2e56dbdb1674badf6295e275"} Nov 02 07:19:06.083752 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.083469 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:06.233076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.230596 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\") pod \"1887eb19-efa5-429e-bb20-992e7144f6f6\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " Nov 02 07:19:06.233076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.230656 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\") pod \"1887eb19-efa5-429e-bb20-992e7144f6f6\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " Nov 02 07:19:06.233076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.230751 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\" (UniqueName: \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\") pod \"1887eb19-efa5-429e-bb20-992e7144f6f6\" (UID: \"1887eb19-efa5-429e-bb20-992e7144f6f6\") " Nov 02 07:19:06.233076 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.231645 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca" pod "1887eb19-efa5-429e-bb20-992e7144f6f6" (UID: "1887eb19-efa5-429e-bb20-992e7144f6f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:19:06.235578 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.235446 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca" pod "1887eb19-efa5-429e-bb20-992e7144f6f6" (UID: "1887eb19-efa5-429e-bb20-992e7144f6f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:06.235806 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.235775 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l" pod "1887eb19-efa5-429e-bb20-992e7144f6f6" (UID: "1887eb19-efa5-429e-bb20-992e7144f6f6"). InnerVolumeSpecName "kube-api-access-t2b8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:19:06.332413 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.332327 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2b8l\" (UniqueName: \"kubernetes.io/projected/1887eb19-efa5-429e-bb20-992e7144f6f6-kube-api-access-t2b8l\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:06.332413 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.332363 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:06.332413 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.332379 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1887eb19-efa5-429e-bb20-992e7144f6f6-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:06.796010 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.793751 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" event={"ID":"1887eb19-efa5-429e-bb20-992e7144f6f6","Type":"ContainerDied","Data":"853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5"} Nov 02 07:19:06.796010 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.793815 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="853c6b318469c24ae0452c2393ebfc2ea460c746a490449d04c09ddd51004cc5" Nov 02 07:19:06.796010 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:06.793890 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm" Nov 02 07:19:08.555774 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:19:08.552973 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:19:13.841837 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:13.841800 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" podUID="c315397d-390e-4515-add7-628d73fae500" containerName="main" containerID="cri-o://318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7" gracePeriod=45 Nov 02 07:19:14.082899 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.082875 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:19:14.100710 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100657 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.100802 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100751 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.100869 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100804 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\" (UniqueName: \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.100908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100882 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\" (UniqueName: \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.100961 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100912 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.100961 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100923 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:14.101097 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.100984 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\") pod \"c315397d-390e-4515-add7-628d73fae500\" (UID: \"c315397d-390e-4515-add7-628d73fae500\") " Nov 02 07:19:14.101097 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.101028 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:14.101277 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.101256 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:14.101396 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.101377 6273 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-tmp-dir\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.101456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.101395 6273 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-model-cache\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.101456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.101409 6273 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-home\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.102952 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.102931 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:14.103047 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.102970 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "kube-api-access-bwq8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:19:14.104360 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.104334 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm" pod "c315397d-390e-4515-add7-628d73fae500" (UID: "c315397d-390e-4515-add7-628d73fae500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:19:14.202346 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.202319 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq8w\" (UniqueName: \"kubernetes.io/projected/c315397d-390e-4515-add7-628d73fae500-kube-api-access-bwq8w\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.202346 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.202348 6273 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c315397d-390e-4515-add7-628d73fae500-dshm\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.202490 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.202360 6273 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c315397d-390e-4515-add7-628d73fae500-tls-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:14.833548 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.833515 6273 generic.go:356] "Generic (PLEG): container finished" podID="c315397d-390e-4515-add7-628d73fae500" containerID="318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7" exitCode=0 Nov 02 07:19:14.833721 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.833558 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" event={"ID":"c315397d-390e-4515-add7-628d73fae500","Type":"ContainerDied","Data":"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7"} Nov 02 07:19:14.833721 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.833561 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" Nov 02 07:19:14.833721 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.833590 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75" event={"ID":"c315397d-390e-4515-add7-628d73fae500","Type":"ContainerDied","Data":"bb9494213dabc99c90d5697214db6d7b2fc8e5f435b8b138ed6a9640a73cc35b"} Nov 02 07:19:14.833721 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.833606 6273 scope.go:122] "RemoveContainer" containerID="318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7" Nov 02 07:19:14.842084 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.842063 6273 scope.go:122] "RemoveContainer" containerID="318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7" Nov 02 07:19:14.842395 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:19:14.842354 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7\": container with ID starting with 318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7 not found: ID does not exist" containerID="318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7" Nov 02 07:19:14.842438 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.842385 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7"} err="failed to get container status \"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7\": rpc error: code = NotFound desc = could not find container \"318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7\": container with ID starting with 318c6032eea25d8303a3785554a2182ac338550d27db96b090a7f820d4d269a7 not found: ID does not exist" Nov 02 07:19:14.986003 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:14.985964 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:19:15.032165 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:15.032141 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7d585e03-kserve-5949bbb7b9-fwd75"] Nov 02 07:19:16.178486 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:16.178454 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c315397d-390e-4515-add7-628d73fae500" path="/var/lib/kubelet/pods/c315397d-390e-4515-add7-628d73fae500/volumes" Nov 02 07:19:16.644869 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:16.644769 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:19:26.369229 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:26.369197 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:19:27.840338 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:27.840311 6273 scope.go:122] "RemoveContainer" containerID="03131d5e06e150331a1bdf10431314e146b5005d65170de24adc3670da4ca2cf" Nov 02 07:19:35.544694 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.544661 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p"] Nov 02 07:19:35.545324 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545304 6273 cpu_manager.go:414] "containerMap: removing container" podUID="1887eb19-efa5-429e-bb20-992e7144f6f6" containerName="revoke-keys" Nov 02 07:19:35.545419 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545326 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1887eb19-efa5-429e-bb20-992e7144f6f6" containerName="revoke-keys" Nov 02 07:19:35.545419 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545347 6273 cpu_manager.go:414] "containerMap: removing container" podUID="c315397d-390e-4515-add7-628d73fae500" containerName="main" Nov 02 07:19:35.545419 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545357 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c315397d-390e-4515-add7-628d73fae500" containerName="main" Nov 02 07:19:35.545600 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545479 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="1887eb19-efa5-429e-bb20-992e7144f6f6" containerName="revoke-keys" Nov 02 07:19:35.545600 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.545493 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="c315397d-390e-4515-add7-628d73fae500" containerName="main" Nov 02 07:19:35.550332 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.550308 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.600239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.600145 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:19:35.600239 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.600222 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:19:35.601478 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.601455 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p"] Nov 02 07:19:35.646047 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.646001 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:19:35.646047 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.646009 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:19:35.647558 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.647539 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.647610 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.647574 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wz6\" (UniqueName: \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.647695 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.647675 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.748306 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.748275 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wz6\" (UniqueName: \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.748441 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.748362 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.748506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.748460 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.749059 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.749037 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.750707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.750686 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.801303 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.801248 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wz6\" (UniqueName: \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\") pod \"maas-api-revoke-sub-cdaa0dea-fbg4p\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:35.863169 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:35.863144 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:36.225646 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:36.225615 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p"] Nov 02 07:19:36.228962 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:19:36.228934 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5cb1ea_da82_4002_98f1_ab675e5451fb.slice/crio-9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5 WatchSource:0}: Error finding container 9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5: Status 404 returned error can't find the container with id 9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5 Nov 02 07:19:36.966575 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:36.966535 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" event={"ID":"cd5cb1ea-da82-4002-98f1-ab675e5451fb","Type":"ContainerStarted","Data":"d7d45e32c4f8b1c616df3708acac9dfa9cd886d25d9f69cb80a105e8f83b0c63"} Nov 02 07:19:36.967052 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:36.966595 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" event={"ID":"cd5cb1ea-da82-4002-98f1-ab675e5451fb","Type":"ContainerStarted","Data":"9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5"} Nov 02 07:19:37.073485 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:37.073437 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" podStartSLOduration=2.073424799 podStartE2EDuration="2.073424799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:19:37.07219019 +0000 UTC m=+3731.045524873" watchObservedRunningTime="2026-11-02 07:19:37.073424799 +0000 UTC m=+3731.046759482" Nov 02 07:19:37.973274 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:37.973239 6273 generic.go:356] "Generic (PLEG): container finished" podID="cd5cb1ea-da82-4002-98f1-ab675e5451fb" containerID="d7d45e32c4f8b1c616df3708acac9dfa9cd886d25d9f69cb80a105e8f83b0c63" exitCode=0 Nov 02 07:19:37.973665 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:37.973321 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" event={"ID":"cd5cb1ea-da82-4002-98f1-ab675e5451fb","Type":"ContainerDied","Data":"d7d45e32c4f8b1c616df3708acac9dfa9cd886d25d9f69cb80a105e8f83b0c63"} Nov 02 07:19:38.554972 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:19:38.554938 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:19:39.102398 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.102372 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:39.187473 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.187444 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\") pod \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " Nov 02 07:19:39.187624 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.187507 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\") pod \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " Nov 02 07:19:39.187624 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.187559 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\" (UniqueName: \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\") pod \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\" (UID: \"cd5cb1ea-da82-4002-98f1-ab675e5451fb\") " Nov 02 07:19:39.187756 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.187736 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca" pod "cd5cb1ea-da82-4002-98f1-ab675e5451fb" (UID: "cd5cb1ea-da82-4002-98f1-ab675e5451fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:19:39.187874 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.187860 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-openshift-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:39.189498 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.189479 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca" pod "cd5cb1ea-da82-4002-98f1-ab675e5451fb" (UID: "cd5cb1ea-da82-4002-98f1-ab675e5451fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:39.189606 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.189593 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6" pod "cd5cb1ea-da82-4002-98f1-ab675e5451fb" (UID: "cd5cb1ea-da82-4002-98f1-ab675e5451fb"). InnerVolumeSpecName "kube-api-access-w6wz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:19:39.289131 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.289071 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wz6\" (UniqueName: \"kubernetes.io/projected/cd5cb1ea-da82-4002-98f1-ab675e5451fb-kube-api-access-w6wz6\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:39.289131 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.289095 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5cb1ea-da82-4002-98f1-ab675e5451fb-maas-api-xks-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:39.983648 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.983621 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" Nov 02 07:19:39.983826 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.983658 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p" event={"ID":"cd5cb1ea-da82-4002-98f1-ab675e5451fb","Type":"ContainerDied","Data":"9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5"} Nov 02 07:19:39.983826 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:39.983702 6273 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9139a43ac2e09b57505f4e40627c750d45a7427177adb9710b5ca6390da060f5" Nov 02 07:19:46.043877 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:46.043834 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x"] Nov 02 07:19:46.090429 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:46.090393 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9fj2x"] Nov 02 07:19:46.178891 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:46.178860 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed2087a-cea0-453f-82cb-eb7853dc9872" path="/var/lib/kubelet/pods/aed2087a-cea0-453f-82cb-eb7853dc9872/volumes" Nov 02 07:19:48.438679 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.438645 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:19:48.792804 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.792712 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:19:48.793423 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.793399 6273 cpu_manager.go:414] "containerMap: removing container" podUID="cd5cb1ea-da82-4002-98f1-ab675e5451fb" containerName="revoke-keys" Nov 02 07:19:48.793423 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.793420 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5cb1ea-da82-4002-98f1-ab675e5451fb" containerName="revoke-keys" Nov 02 07:19:48.793663 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.793559 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5cb1ea-da82-4002-98f1-ab675e5451fb" containerName="revoke-keys" Nov 02 07:19:48.798341 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.798315 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.810936 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.810883 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4"] Nov 02 07:19:48.815477 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.815453 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.854905 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.854875 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:19:48.855056 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.854977 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 07:19:48.856354 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.856288 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:19:48.878492 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878471 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.878602 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878503 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.878602 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878540 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nh8l\" (UniqueName: \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.878772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878662 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclt6\" (UniqueName: \"kubernetes.io/projected/032eef7f-ff8a-4d63-b409-47270dbf4f17-kube-api-access-hclt6\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.878772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878691 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-metrics-certs\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.878772 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878746 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.878934 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878892 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-maas-api-tls\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.879025 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.878961 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/032eef7f-ff8a-4d63-b409-47270dbf4f17-openshift-service-ca\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.894271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.894250 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:19:48.894541 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.894519 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 07:19:48.894639 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.894552 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 07:19:48.896872 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.896849 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4"] Nov 02 07:19:48.980022 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.979979 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-metrics-certs\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.980187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980042 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.980187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980087 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-maas-api-tls\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.980187 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980141 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/032eef7f-ff8a-4d63-b409-47270dbf4f17-openshift-service-ca\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.980362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980224 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.980362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980257 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.980362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980301 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nh8l\" (UniqueName: \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.980362 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.980343 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclt6\" (UniqueName: \"kubernetes.io/projected/032eef7f-ff8a-4d63-b409-47270dbf4f17-kube-api-access-hclt6\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.981054 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.981032 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/032eef7f-ff8a-4d63-b409-47270dbf4f17-openshift-service-ca\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.981212 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.981189 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.982707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.982686 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-maas-api-tls\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.982847 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.982826 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032eef7f-ff8a-4d63-b409-47270dbf4f17-metrics-certs\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:48.982926 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.982887 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:48.983046 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:48.983025 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:49.033583 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.033561 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclt6\" (UniqueName: \"kubernetes.io/projected/032eef7f-ff8a-4d63-b409-47270dbf4f17-kube-api-access-hclt6\") pod \"maas-api-5fbfc94f97-wfbt4\" (UID: \"032eef7f-ff8a-4d63-b409-47270dbf4f17\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:49.033688 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.033587 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nh8l\" (UniqueName: \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\") pod \"maas-api-7dc858b877-7xxph\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:49.114913 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.114861 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:49.128065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.128043 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:49.496480 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.496441 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:19:49.521237 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:19:49.521208 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f811ba2_8203_4503_b84b_c018ef424eb4.slice/crio-22368644fed1aad2d2ee02e3cab0d5803c428749ac7486ced8c360e90e411eec WatchSource:0}: Error finding container 22368644fed1aad2d2ee02e3cab0d5803c428749ac7486ced8c360e90e411eec: Status 404 returned error can't find the container with id 22368644fed1aad2d2ee02e3cab0d5803c428749ac7486ced8c360e90e411eec Nov 02 07:19:49.522547 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.522529 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:19:49.529248 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:49.529221 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4"] Nov 02 07:19:49.536553 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:19:49.536533 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032eef7f_ff8a_4d63_b409_47270dbf4f17.slice/crio-952b16fba3b1eb12c573682cbc6882a1932ff708c209b504af50e2e5ae2117d6 WatchSource:0}: Error finding container 952b16fba3b1eb12c573682cbc6882a1932ff708c209b504af50e2e5ae2117d6: Status 404 returned error can't find the container with id 952b16fba3b1eb12c573682cbc6882a1932ff708c209b504af50e2e5ae2117d6 Nov 02 07:19:50.038174 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:50.038136 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" event={"ID":"6f811ba2-8203-4503-b84b-c018ef424eb4","Type":"ContainerStarted","Data":"22368644fed1aad2d2ee02e3cab0d5803c428749ac7486ced8c360e90e411eec"} Nov 02 07:19:50.039219 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:50.039193 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" event={"ID":"032eef7f-ff8a-4d63-b409-47270dbf4f17","Type":"ContainerStarted","Data":"952b16fba3b1eb12c573682cbc6882a1932ff708c209b504af50e2e5ae2117d6"} Nov 02 07:19:51.045360 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.045318 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" event={"ID":"6f811ba2-8203-4503-b84b-c018ef424eb4","Type":"ContainerStarted","Data":"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee"} Nov 02 07:19:51.045821 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.045393 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:51.046911 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.046888 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" event={"ID":"032eef7f-ff8a-4d63-b409-47270dbf4f17","Type":"ContainerStarted","Data":"4f4d3d80540decaca5aa66fd9d872433a875d75bffc7457af60f277a82bb1f24"} Nov 02 07:19:51.047038 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.046931 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:51.261534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.261479 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" podStartSLOduration=2.205088281 podStartE2EDuration="3.261464384s" totalImagesPullingTime="1.056376103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:19:51.152644366 +0000 UTC m=+3745.125979096" watchObservedRunningTime="2026-11-02 07:19:51.261464384 +0000 UTC m=+3745.234799073" Nov 02 07:19:51.262563 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:51.262525 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" podStartSLOduration=2.3571937370000002 podStartE2EDuration="3.262514699s" totalImagesPullingTime="905.320962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:19:51.260329331 +0000 UTC m=+3745.233664015" watchObservedRunningTime="2026-11-02 07:19:51.262514699 +0000 UTC m=+3745.235849382" Nov 02 07:19:57.057195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.057165 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-wfbt4" Nov 02 07:19:57.057801 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.057687 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:57.211758 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.211728 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:19:57.212098 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.211886 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" podUID="6f811ba2-8203-4503-b84b-c018ef424eb4" containerName="maas-api" containerID="cri-o://66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee" gracePeriod=30 Nov 02 07:19:57.473915 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.473886 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:57.561128 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.561097 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\") pod \"6f811ba2-8203-4503-b84b-c018ef424eb4\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " Nov 02 07:19:57.561286 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.561178 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\" (UniqueName: \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\") pod \"6f811ba2-8203-4503-b84b-c018ef424eb4\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " Nov 02 07:19:57.561286 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.561239 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\") pod \"6f811ba2-8203-4503-b84b-c018ef424eb4\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " Nov 02 07:19:57.561286 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.561277 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\") pod \"6f811ba2-8203-4503-b84b-c018ef424eb4\" (UID: \"6f811ba2-8203-4503-b84b-c018ef424eb4\") " Nov 02 07:19:57.561711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.561685 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca" pod "6f811ba2-8203-4503-b84b-c018ef424eb4" (UID: "6f811ba2-8203-4503-b84b-c018ef424eb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:19:57.563372 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.563344 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs" pod "6f811ba2-8203-4503-b84b-c018ef424eb4" (UID: "6f811ba2-8203-4503-b84b-c018ef424eb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:57.563372 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.563364 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls" pod "6f811ba2-8203-4503-b84b-c018ef424eb4" (UID: "6f811ba2-8203-4503-b84b-c018ef424eb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:19:57.563518 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.563442 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l" pod "6f811ba2-8203-4503-b84b-c018ef424eb4" (UID: "6f811ba2-8203-4503-b84b-c018ef424eb4"). InnerVolumeSpecName "kube-api-access-6nh8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:19:57.662382 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.662355 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:57.662382 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.662381 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nh8l\" (UniqueName: \"kubernetes.io/projected/6f811ba2-8203-4503-b84b-c018ef424eb4-kube-api-access-6nh8l\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:57.662564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.662392 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f811ba2-8203-4503-b84b-c018ef424eb4-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:57.662564 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:57.662402 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f811ba2-8203-4503-b84b-c018ef424eb4-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:19:58.080887 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.080778 6273 generic.go:356] "Generic (PLEG): container finished" podID="6f811ba2-8203-4503-b84b-c018ef424eb4" containerID="66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee" exitCode=0 Nov 02 07:19:58.080887 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.080849 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" Nov 02 07:19:58.081370 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.080877 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" event={"ID":"6f811ba2-8203-4503-b84b-c018ef424eb4","Type":"ContainerDied","Data":"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee"} Nov 02 07:19:58.081370 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.080938 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-7xxph" event={"ID":"6f811ba2-8203-4503-b84b-c018ef424eb4","Type":"ContainerDied","Data":"22368644fed1aad2d2ee02e3cab0d5803c428749ac7486ced8c360e90e411eec"} Nov 02 07:19:58.081370 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.080961 6273 scope.go:122] "RemoveContainer" containerID="66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee" Nov 02 07:19:58.089867 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.089848 6273 scope.go:122] "RemoveContainer" containerID="66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee" Nov 02 07:19:58.090189 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:19:58.090171 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee\": container with ID starting with 66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee not found: ID does not exist" containerID="66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee" Nov 02 07:19:58.090247 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.090197 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee"} err="failed to get container status \"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee\": rpc error: code = NotFound desc = could not find container \"66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee\": container with ID starting with 66cc36ac6b2853c69d93cc816ce72039b89ad65065a3a131db0772f9a907faee not found: ID does not exist" Nov 02 07:19:58.233015 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.232978 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:19:58.236190 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:19:58.236169 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-7xxph"] Nov 02 07:20:00.179155 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:00.179118 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f811ba2-8203-4503-b84b-c018ef424eb4" path="/var/lib/kubelet/pods/6f811ba2-8203-4503-b84b-c018ef424eb4/volumes" Nov 02 07:20:02.888034 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.888001 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:02.888436 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.888423 6273 cpu_manager.go:414] "containerMap: removing container" podUID="6f811ba2-8203-4503-b84b-c018ef424eb4" containerName="maas-api" Nov 02 07:20:02.888480 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.888437 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f811ba2-8203-4503-b84b-c018ef424eb4" containerName="maas-api" Nov 02 07:20:02.888521 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.888500 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f811ba2-8203-4503-b84b-c018ef424eb4" containerName="maas-api" Nov 02 07:20:02.892737 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.892716 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:02.916167 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.916144 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:02.919547 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.919533 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:02.959733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.959701 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e3c0bf7-serving-cert\"" Nov 02 07:20:02.961374 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.961343 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:02.992245 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.992213 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e3c0bf7-dockercfg-6k8rb\"" Nov 02 07:20:02.992245 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.992218 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e3c0bf7\"" Nov 02 07:20:02.994165 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:02.994143 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:03.009738 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.009712 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njx6\" (UniqueName: \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.009852 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.009765 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.009930 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.009857 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdk82\" (UniqueName: \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.010019 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.009951 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.010019 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.009981 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.010152 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.010096 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.063711 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.063678 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:03.067272 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.067253 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.110940 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.110913 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.111072 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.110958 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdk82\" (UniqueName: \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.111072 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111025 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.111072 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111050 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jg5f\" (UniqueName: \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.111287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111077 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.111287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111130 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.111287 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111242 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.111448 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:20:03.111287 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-serving-cert: secret "maas-api-e2e-ait-0e3c0bf7-serving-cert" not found Nov 02 07:20:03.111448 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111317 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njx6\" (UniqueName: \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.111448 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:20:03.111369 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls podName:2cd81a78-98a3-465d-ad26-22a0dd7fe7c0 nodeName:}" failed. No retries permitted until 2026-11-02 07:20:03.611340424 +0000 UTC m=+3757.584675085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls") pod "maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" (UID: "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0") : secret "maas-api-e2e-ait-0e3c0bf7-serving-cert" not found Nov 02 07:20:03.111802 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111781 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.111858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.111844 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.113329 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.113307 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.122186 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.122165 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:03.170373 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.170350 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njx6\" (UniqueName: \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.170647 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.170624 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdk82\" (UniqueName: \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\") pod \"payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.212313 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.212282 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jg5f\" (UniqueName: \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.212456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.212369 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.213086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.213065 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.231481 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.231455 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:03.265750 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.265717 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jg5f\" (UniqueName: \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\") pod \"payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.379858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.379825 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:03.602176 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.602142 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:03.605497 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:20:03.605472 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda714ed5_f6d0_44f7_b070_12d099cbae47.slice/crio-a0dd354a46d6e75924a0b8b057566dd7a24de47f5d479e867e29878523a37304 WatchSource:0}: Error finding container a0dd354a46d6e75924a0b8b057566dd7a24de47f5d479e867e29878523a37304: Status 404 returned error can't find the container with id a0dd354a46d6e75924a0b8b057566dd7a24de47f5d479e867e29878523a37304 Nov 02 07:20:03.619154 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.619133 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.621314 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.621291 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") pod \"maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:03.745705 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.745679 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:03.748082 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:20:03.748054 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a671859_78e0_4fc5_a4bf_266e891f9bec.slice/crio-74590d03766c83d66761e48574fe8452e9706211c1625a2cbf30bfcf5e0bc56a WatchSource:0}: Error finding container 74590d03766c83d66761e48574fe8452e9706211c1625a2cbf30bfcf5e0bc56a: Status 404 returned error can't find the container with id 74590d03766c83d66761e48574fe8452e9706211c1625a2cbf30bfcf5e0bc56a Nov 02 07:20:03.805289 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:03.805271 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:04.112843 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.112740 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" event={"ID":"da714ed5-f6d0-44f7-b070-12d099cbae47","Type":"ContainerStarted","Data":"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e"} Nov 02 07:20:04.112843 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.112798 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" event={"ID":"da714ed5-f6d0-44f7-b070-12d099cbae47","Type":"ContainerStarted","Data":"a0dd354a46d6e75924a0b8b057566dd7a24de47f5d479e867e29878523a37304"} Nov 02 07:20:04.113385 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.112950 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:04.114751 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.114726 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" event={"ID":"7a671859-78e0-4fc5-a4bf-266e891f9bec","Type":"ContainerStarted","Data":"c6686ccf9150502a55c42c455de046f6c3a32446050aabbffbb36c1574ea5769"} Nov 02 07:20:04.114893 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.114764 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" event={"ID":"7a671859-78e0-4fc5-a4bf-266e891f9bec","Type":"ContainerStarted","Data":"74590d03766c83d66761e48574fe8452e9706211c1625a2cbf30bfcf5e0bc56a"} Nov 02 07:20:04.114893 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.114783 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:04.172135 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.172109 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:04.175790 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:20:04.175762 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd81a78_98a3_465d_ad26_22a0dd7fe7c0.slice/crio-f5f7c587dfeafef19436146fb84efce199898785fb03efee745aef08eee9a313 WatchSource:0}: Error finding container f5f7c587dfeafef19436146fb84efce199898785fb03efee745aef08eee9a313: Status 404 returned error can't find the container with id f5f7c587dfeafef19436146fb84efce199898785fb03efee745aef08eee9a313 Nov 02 07:20:04.221660 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.221617 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" podStartSLOduration=2.221603957 podStartE2EDuration="2.221603957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:20:04.220454542 +0000 UTC m=+3758.193789226" watchObservedRunningTime="2026-11-02 07:20:04.221603957 +0000 UTC m=+3758.194938637" Nov 02 07:20:04.329720 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:04.329662 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" podStartSLOduration=1.329646139 podStartE2EDuration="1.329646139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:20:04.329022522 +0000 UTC m=+3758.302357207" watchObservedRunningTime="2026-11-02 07:20:04.329646139 +0000 UTC m=+3758.302980823" Nov 02 07:20:05.123382 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:05.123356 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" event={"ID":"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0","Type":"ContainerStarted","Data":"f5f7c587dfeafef19436146fb84efce199898785fb03efee745aef08eee9a313"} Nov 02 07:20:06.130304 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:06.130273 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" event={"ID":"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0","Type":"ContainerStarted","Data":"2d35141d8af1ddf4c3b5f3579315a1162b0bf0033ff2193cb2997d5ef48a3d9b"} Nov 02 07:20:06.130750 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:06.130342 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:06.239130 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:06.239081 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" podStartSLOduration=3.325765375 podStartE2EDuration="4.239068983s" totalImagesPullingTime="913.303608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:20:06.237374682 +0000 UTC m=+3760.210709368" watchObservedRunningTime="2026-11-02 07:20:06.239068983 +0000 UTC m=+3760.212403662" Nov 02 07:20:08.552347 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:20:08.552313 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:20:11.391672 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:11.391638 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:20:12.140344 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:12.140315 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:15.126534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:15.126500 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:15.126935 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:15.126755 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:38.551776 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:20:38.551706 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:20:42.140126 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:42.140091 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:42.239235 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:42.239204 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:47.140962 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.140919 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" podUID="7a671859-78e0-4fc5-a4bf-266e891f9bec" containerName="payload-processing" containerID="cri-o://c6686ccf9150502a55c42c455de046f6c3a32446050aabbffbb36c1574ea5769" gracePeriod=25 Nov 02 07:20:47.239640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.239605 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" podUID="da714ed5-f6d0-44f7-b070-12d099cbae47" containerName="payload-pre-processing" containerID="cri-o://4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e" gracePeriod=25 Nov 02 07:20:47.327401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.327371 6273 generic.go:356] "Generic (PLEG): container finished" podID="7a671859-78e0-4fc5-a4bf-266e891f9bec" containerID="c6686ccf9150502a55c42c455de046f6c3a32446050aabbffbb36c1574ea5769" exitCode=0 Nov 02 07:20:47.327555 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.327442 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" event={"ID":"7a671859-78e0-4fc5-a4bf-266e891f9bec","Type":"ContainerDied","Data":"c6686ccf9150502a55c42c455de046f6c3a32446050aabbffbb36c1574ea5769"} Nov 02 07:20:47.412047 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.412018 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:47.512606 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.512583 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:47.516638 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.516613 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\") pod \"7a671859-78e0-4fc5-a4bf-266e891f9bec\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " Nov 02 07:20:47.516749 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.516689 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\" (UniqueName: \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\") pod \"7a671859-78e0-4fc5-a4bf-266e891f9bec\" (UID: \"7a671859-78e0-4fc5-a4bf-266e891f9bec\") " Nov 02 07:20:47.517118 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.517087 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume" pod "7a671859-78e0-4fc5-a4bf-266e891f9bec" (UID: "7a671859-78e0-4fc5-a4bf-266e891f9bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:47.517221 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.517207 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a671859-78e0-4fc5-a4bf-266e891f9bec-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:47.518800 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.518774 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f" pod "7a671859-78e0-4fc5-a4bf-266e891f9bec" (UID: "7a671859-78e0-4fc5-a4bf-266e891f9bec"). InnerVolumeSpecName "kube-api-access-5jg5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:47.618342 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.618309 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\" (UniqueName: \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\") pod \"da714ed5-f6d0-44f7-b070-12d099cbae47\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " Nov 02 07:20:47.618534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.618362 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\") pod \"da714ed5-f6d0-44f7-b070-12d099cbae47\" (UID: \"da714ed5-f6d0-44f7-b070-12d099cbae47\") " Nov 02 07:20:47.618645 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.618631 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jg5f\" (UniqueName: \"kubernetes.io/projected/7a671859-78e0-4fc5-a4bf-266e891f9bec-kube-api-access-5jg5f\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:47.618718 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.618690 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume" pod "da714ed5-f6d0-44f7-b070-12d099cbae47" (UID: "da714ed5-f6d0-44f7-b070-12d099cbae47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:47.620383 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.620353 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82" pod "da714ed5-f6d0-44f7-b070-12d099cbae47" (UID: "da714ed5-f6d0-44f7-b070-12d099cbae47"). InnerVolumeSpecName "kube-api-access-qdk82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:47.719111 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.719088 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdk82\" (UniqueName: \"kubernetes.io/projected/da714ed5-f6d0-44f7-b070-12d099cbae47-kube-api-access-qdk82\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:47.719111 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:47.719111 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da714ed5-f6d0-44f7-b070-12d099cbae47-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:48.113808 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.113733 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:48.113966 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.113914 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" podUID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" containerName="maas-api" containerID="cri-o://2d35141d8af1ddf4c3b5f3579315a1162b0bf0033ff2193cb2997d5ef48a3d9b" gracePeriod=30 Nov 02 07:20:48.336242 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.336210 6273 generic.go:356] "Generic (PLEG): container finished" podID="da714ed5-f6d0-44f7-b070-12d099cbae47" containerID="4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e" exitCode=0 Nov 02 07:20:48.336717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.336250 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" Nov 02 07:20:48.336717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.336296 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" event={"ID":"da714ed5-f6d0-44f7-b070-12d099cbae47","Type":"ContainerDied","Data":"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e"} Nov 02 07:20:48.336717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.336357 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd" event={"ID":"da714ed5-f6d0-44f7-b070-12d099cbae47","Type":"ContainerDied","Data":"a0dd354a46d6e75924a0b8b057566dd7a24de47f5d479e867e29878523a37304"} Nov 02 07:20:48.336717 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.336387 6273 scope.go:122] "RemoveContainer" containerID="4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e" Nov 02 07:20:48.337911 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.337883 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" Nov 02 07:20:48.338171 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.337926 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws" event={"ID":"7a671859-78e0-4fc5-a4bf-266e891f9bec","Type":"ContainerDied","Data":"74590d03766c83d66761e48574fe8452e9706211c1625a2cbf30bfcf5e0bc56a"} Nov 02 07:20:48.340039 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.340015 6273 generic.go:356] "Generic (PLEG): container finished" podID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" containerID="2d35141d8af1ddf4c3b5f3579315a1162b0bf0033ff2193cb2997d5ef48a3d9b" exitCode=0 Nov 02 07:20:48.340136 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.340055 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" event={"ID":"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0","Type":"ContainerDied","Data":"2d35141d8af1ddf4c3b5f3579315a1162b0bf0033ff2193cb2997d5ef48a3d9b"} Nov 02 07:20:48.346969 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.346952 6273 scope.go:122] "RemoveContainer" containerID="4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e" Nov 02 07:20:48.347421 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:20:48.347394 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e\": container with ID starting with 4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e not found: ID does not exist" containerID="4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e" Nov 02 07:20:48.347528 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.347431 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e"} err="failed to get container status \"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e\": rpc error: code = NotFound desc = could not find container \"4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e\": container with ID starting with 4fc501f74c605a7c93258fe4cd918dfd41fa072100600b238dee4f0d4eb36f8e not found: ID does not exist" Nov 02 07:20:48.347528 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.347454 6273 scope.go:122] "RemoveContainer" containerID="c6686ccf9150502a55c42c455de046f6c3a32446050aabbffbb36c1574ea5769" Nov 02 07:20:48.372071 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.372048 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:48.425640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.425617 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") pod \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " Nov 02 07:20:48.425719 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.425667 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\") pod \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " Nov 02 07:20:48.425771 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.425744 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\" (UniqueName: \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\") pod \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " Nov 02 07:20:48.425853 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.425835 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\") pod \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\" (UID: \"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0\") " Nov 02 07:20:48.426087 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.426061 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca" pod "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" (UID: "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:48.426218 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.426197 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:48.427490 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.427471 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls" pod "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" (UID: "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:20:48.427683 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.427665 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6" pod "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" (UID: "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0"). InnerVolumeSpecName "kube-api-access-6njx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:48.427848 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.427834 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs" pod "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" (UID: "2cd81a78-98a3-465d-ad26-22a0dd7fe7c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:20:48.488495 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.488463 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:48.491459 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.491427 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3c0bf7-79c4698f9-sv4cd"] Nov 02 07:20:48.527257 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.527228 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:48.527257 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.527253 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:48.527413 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.527265 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6njx6\" (UniqueName: \"kubernetes.io/projected/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0-kube-api-access-6njx6\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:48.592612 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.592582 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:48.596401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:48.596372 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3c0bf7-54589967ff-b9fws"] Nov 02 07:20:49.347930 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:49.347896 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" event={"ID":"2cd81a78-98a3-465d-ad26-22a0dd7fe7c0","Type":"ContainerDied","Data":"f5f7c587dfeafef19436146fb84efce199898785fb03efee745aef08eee9a313"} Nov 02 07:20:49.348391 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:49.347961 6273 scope.go:122] "RemoveContainer" containerID="2d35141d8af1ddf4c3b5f3579315a1162b0bf0033ff2193cb2997d5ef48a3d9b" Nov 02 07:20:49.348391 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:49.347908 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm" Nov 02 07:20:49.499858 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:49.499831 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:49.507581 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:49.507560 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3c0bf7-586dcd5989-hfbzm"] Nov 02 07:20:50.178051 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:50.178003 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" path="/var/lib/kubelet/pods/2cd81a78-98a3-465d-ad26-22a0dd7fe7c0/volumes" Nov 02 07:20:50.178408 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:50.178394 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a671859-78e0-4fc5-a4bf-266e891f9bec" path="/var/lib/kubelet/pods/7a671859-78e0-4fc5-a4bf-266e891f9bec/volumes" Nov 02 07:20:50.178731 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:50.178720 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da714ed5-f6d0-44f7-b070-12d099cbae47" path="/var/lib/kubelet/pods/da714ed5-f6d0-44f7-b070-12d099cbae47/volumes" Nov 02 07:20:54.784763 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:20:54.784727 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:21:01.748353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748322 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:21:01.748762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748717 6273 cpu_manager.go:414] "containerMap: removing container" podUID="7a671859-78e0-4fc5-a4bf-266e891f9bec" containerName="payload-processing" Nov 02 07:21:01.748762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748728 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a671859-78e0-4fc5-a4bf-266e891f9bec" containerName="payload-processing" Nov 02 07:21:01.748762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748741 6273 cpu_manager.go:414] "containerMap: removing container" podUID="da714ed5-f6d0-44f7-b070-12d099cbae47" containerName="payload-pre-processing" Nov 02 07:21:01.748762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748746 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da714ed5-f6d0-44f7-b070-12d099cbae47" containerName="payload-pre-processing" Nov 02 07:21:01.748762 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748763 6273 cpu_manager.go:414] "containerMap: removing container" podUID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" containerName="maas-api" Nov 02 07:21:01.748960 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748767 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" containerName="maas-api" Nov 02 07:21:01.748960 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748891 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a671859-78e0-4fc5-a4bf-266e891f9bec" containerName="payload-processing" Nov 02 07:21:01.748960 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748900 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="da714ed5-f6d0-44f7-b070-12d099cbae47" containerName="payload-pre-processing" Nov 02 07:21:01.748960 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.748908 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd81a78-98a3-465d-ad26-22a0dd7fe7c0" containerName="maas-api" Nov 02 07:21:01.752162 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.752142 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.848265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.848228 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-c0d3ab-dockercfg-5lh67\"" Nov 02 07:21:01.848265 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.848261 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-c0d3ab\"" Nov 02 07:21:01.850058 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.850028 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:21:01.852277 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.852257 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.852383 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.852337 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtz7w\" (UniqueName: \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.911854 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.911828 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:21:01.915401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.915383 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:01.953119 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.953092 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.953215 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.953177 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtz7w\" (UniqueName: \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.953787 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.953766 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:01.968340 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:01.968309 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:21:02.006435 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.006382 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtz7w\" (UniqueName: \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\") pod \"payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:02.054065 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.054022 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.054214 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.054189 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b494\" (UniqueName: \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.065136 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.065115 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:02.155212 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.155185 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.155387 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.155253 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b494\" (UniqueName: \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.155851 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.155828 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.208727 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.208698 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b494\" (UniqueName: \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\") pod \"payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.228597 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.228572 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:02.428534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.428508 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:21:02.431113 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:21:02.431087 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bb5e72_b82f_4cc3_96e5_4a432ac38215.slice/crio-a663134aee554d299112f15c6e849caff178e9197e74037a923c1a591d0cb116 WatchSource:0}: Error finding container a663134aee554d299112f15c6e849caff178e9197e74037a923c1a591d0cb116: Status 404 returned error can't find the container with id a663134aee554d299112f15c6e849caff178e9197e74037a923c1a591d0cb116 Nov 02 07:21:02.598342 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:02.598270 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:21:02.601208 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:21:02.601180 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52a2957_ccc4_4921_bec4_552d78bc029a.slice/crio-7573269487d689d6bbe13d2f032519bfa273cc9d4d4c69758bf4f01b5c465257 WatchSource:0}: Error finding container 7573269487d689d6bbe13d2f032519bfa273cc9d4d4c69758bf4f01b5c465257: Status 404 returned error can't find the container with id 7573269487d689d6bbe13d2f032519bfa273cc9d4d4c69758bf4f01b5c465257 Nov 02 07:21:03.420716 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.420674 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" event={"ID":"a1bb5e72-b82f-4cc3-96e5-4a432ac38215","Type":"ContainerStarted","Data":"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1"} Nov 02 07:21:03.421195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.420740 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" event={"ID":"a1bb5e72-b82f-4cc3-96e5-4a432ac38215","Type":"ContainerStarted","Data":"a663134aee554d299112f15c6e849caff178e9197e74037a923c1a591d0cb116"} Nov 02 07:21:03.421195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.420791 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:03.422301 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.422279 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" event={"ID":"d52a2957-ccc4-4921-bec4-552d78bc029a","Type":"ContainerStarted","Data":"190305ce599cd3ddd29c6275ab803907f0a0a04a5520524df6fd82180e211a9d"} Nov 02 07:21:03.422371 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.422317 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" event={"ID":"d52a2957-ccc4-4921-bec4-552d78bc029a","Type":"ContainerStarted","Data":"7573269487d689d6bbe13d2f032519bfa273cc9d4d4c69758bf4f01b5c465257"} Nov 02 07:21:03.422437 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.422422 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:03.527942 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.527895 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" podStartSLOduration=2.52788336 podStartE2EDuration="2.52788336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:21:03.526487711 +0000 UTC m=+3817.499822395" watchObservedRunningTime="2026-11-02 07:21:03.52788336 +0000 UTC m=+3817.501218042" Nov 02 07:21:03.632447 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.632344 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" podStartSLOduration=2.632328694 podStartE2EDuration="2.632328694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:21:03.631560707 +0000 UTC m=+3817.604895403" watchObservedRunningTime="2026-11-02 07:21:03.632328694 +0000 UTC m=+3817.605663378" Nov 02 07:21:03.992038 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.991983 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:21:03.995936 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:03.995915 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.044405 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.044385 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-c0d3ab-serving-cert\"" Nov 02 07:21:04.073064 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.073035 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.073195 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.073100 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.073249 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.073221 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.073351 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.073325 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr28p\" (UniqueName: \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.092341 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.092317 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:21:04.173884 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.173851 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr28p\" (UniqueName: \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.174086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.173920 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.174086 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.173953 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.174216 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:21:04.174102 6273 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-serving-cert: secret "maas-api-e2e-authcfg-http-c0d3ab-serving-cert" not found Nov 02 07:21:04.174216 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.174098 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.174216 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:21:04.174180 6273 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls podName:a267274d-cb52-4f03-aafa-10cc4a597aae nodeName:}" failed. No retries permitted until 2026-11-02 07:21:04.674158593 +0000 UTC m=+3818.647493265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls") pod "maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" (UID: "a267274d-cb52-4f03-aafa-10cc4a597aae") : secret "maas-api-e2e-authcfg-http-c0d3ab-serving-cert" not found Nov 02 07:21:04.174785 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.174755 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.176716 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.176694 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.227640 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.227620 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr28p\" (UniqueName: \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.679209 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.679180 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.681653 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.681624 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:04.909108 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:04.909076 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:05.272314 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:05.272116 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:21:05.274684 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:21:05.274653 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda267274d_cb52_4f03_aafa_10cc4a597aae.slice/crio-0654f594cd2cf762960d49e7d1cca83ef90adac8558c48fe9320e2deeed6b7e1 WatchSource:0}: Error finding container 0654f594cd2cf762960d49e7d1cca83ef90adac8558c48fe9320e2deeed6b7e1: Status 404 returned error can't find the container with id 0654f594cd2cf762960d49e7d1cca83ef90adac8558c48fe9320e2deeed6b7e1 Nov 02 07:21:05.437084 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:05.437046 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" event={"ID":"a267274d-cb52-4f03-aafa-10cc4a597aae","Type":"ContainerStarted","Data":"0654f594cd2cf762960d49e7d1cca83ef90adac8558c48fe9320e2deeed6b7e1"} Nov 02 07:21:06.443729 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:06.443652 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" event={"ID":"a267274d-cb52-4f03-aafa-10cc4a597aae","Type":"ContainerStarted","Data":"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250"} Nov 02 07:21:06.444132 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:06.443794 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:06.551761 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:06.551703 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" podStartSLOduration=2.635050062 podStartE2EDuration="3.551687173s" totalImagesPullingTime="916.637111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:21:06.551082259 +0000 UTC m=+3820.524416941" watchObservedRunningTime="2026-11-02 07:21:06.551687173 +0000 UTC m=+3820.525021857" Nov 02 07:21:08.552106 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:21:08.552067 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:21:12.157982 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:12.157951 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:21:12.454734 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:12.454660 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:21:14.432621 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:14.432590 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:21:14.433078 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:14.432655 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:21:27.880029 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:21:27.879979 6273 scope.go:122] "RemoveContainer" containerID="ade40f531a536a8b34832ac7a08f180cb0a7a71b7db1b1a2e2198007b9e5bfae" Nov 02 07:21:38.551693 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:21:38.551655 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:22:08.553419 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:22:08.553336 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:22:38.551233 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:22:38.551197 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:22:54.860949 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:54.860914 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:22:54.958325 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:54.958289 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:22:59.861627 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:59.861571 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" podUID="d52a2957-ccc4-4921-bec4-552d78bc029a" containerName="payload-processing" containerID="cri-o://190305ce599cd3ddd29c6275ab803907f0a0a04a5520524df6fd82180e211a9d" gracePeriod=25 Nov 02 07:22:59.959155 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:59.959105 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" podUID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" containerName="payload-pre-processing" containerID="cri-o://0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1" gracePeriod=25 Nov 02 07:22:59.991196 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:59.991170 6273 generic.go:356] "Generic (PLEG): container finished" podID="d52a2957-ccc4-4921-bec4-552d78bc029a" containerID="190305ce599cd3ddd29c6275ab803907f0a0a04a5520524df6fd82180e211a9d" exitCode=0 Nov 02 07:22:59.991302 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:22:59.991258 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" event={"ID":"d52a2957-ccc4-4921-bec4-552d78bc029a","Type":"ContainerDied","Data":"190305ce599cd3ddd29c6275ab803907f0a0a04a5520524df6fd82180e211a9d"} Nov 02 07:23:00.123197 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.123140 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:23:00.211969 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.211941 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\" (UniqueName: \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\") pod \"d52a2957-ccc4-4921-bec4-552d78bc029a\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " Nov 02 07:23:00.212120 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.212025 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\") pod \"d52a2957-ccc4-4921-bec4-552d78bc029a\" (UID: \"d52a2957-ccc4-4921-bec4-552d78bc029a\") " Nov 02 07:23:00.212424 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.212402 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume" pod "d52a2957-ccc4-4921-bec4-552d78bc029a" (UID: "d52a2957-ccc4-4921-bec4-552d78bc029a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:23:00.214029 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.214008 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494" pod "d52a2957-ccc4-4921-bec4-552d78bc029a" (UID: "d52a2957-ccc4-4921-bec4-552d78bc029a"). InnerVolumeSpecName "kube-api-access-4b494". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:00.221167 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.221149 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:23:00.313094 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.313069 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\" (UniqueName: \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\") pod \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " Nov 02 07:23:00.313245 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.313178 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\") pod \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\" (UID: \"a1bb5e72-b82f-4cc3-96e5-4a432ac38215\") " Nov 02 07:23:00.313542 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.313519 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b494\" (UniqueName: \"kubernetes.io/projected/d52a2957-ccc4-4921-bec4-552d78bc029a-kube-api-access-4b494\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:00.313542 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.313539 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52a2957-ccc4-4921-bec4-552d78bc029a-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:00.313542 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.313535 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume" pod "a1bb5e72-b82f-4cc3-96e5-4a432ac38215" (UID: "a1bb5e72-b82f-4cc3-96e5-4a432ac38215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:23:00.315146 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.315126 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w" pod "a1bb5e72-b82f-4cc3-96e5-4a432ac38215" (UID: "a1bb5e72-b82f-4cc3-96e5-4a432ac38215"). InnerVolumeSpecName "kube-api-access-dtz7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:00.414797 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.414774 6273 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-plugins-config-volume\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:00.414797 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.414797 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtz7w\" (UniqueName: \"kubernetes.io/projected/a1bb5e72-b82f-4cc3-96e5-4a432ac38215-kube-api-access-dtz7w\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:00.997384 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.997352 6273 generic.go:356] "Generic (PLEG): container finished" podID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" containerID="0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1" exitCode=0 Nov 02 07:23:00.997784 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.997397 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" Nov 02 07:23:00.997784 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.997433 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" event={"ID":"a1bb5e72-b82f-4cc3-96e5-4a432ac38215","Type":"ContainerDied","Data":"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1"} Nov 02 07:23:00.997784 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.997488 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq" event={"ID":"a1bb5e72-b82f-4cc3-96e5-4a432ac38215","Type":"ContainerDied","Data":"a663134aee554d299112f15c6e849caff178e9197e74037a923c1a591d0cb116"} Nov 02 07:23:00.997784 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.997515 6273 scope.go:122] "RemoveContainer" containerID="0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1" Nov 02 07:23:00.998829 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.998814 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" Nov 02 07:23:00.998928 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:00.998858 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b" event={"ID":"d52a2957-ccc4-4921-bec4-552d78bc029a","Type":"ContainerDied","Data":"7573269487d689d6bbe13d2f032519bfa273cc9d4d4c69758bf4f01b5c465257"} Nov 02 07:23:01.006530 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.006510 6273 scope.go:122] "RemoveContainer" containerID="0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1" Nov 02 07:23:01.006782 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:23:01.006756 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1\": container with ID starting with 0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1 not found: ID does not exist" containerID="0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1" Nov 02 07:23:01.006891 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.006782 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1"} err="failed to get container status \"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1\": rpc error: code = NotFound desc = could not find container \"0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1\": container with ID starting with 0b0684692cacf167aad13a7d7878110eabd16165d05a6001d445f476fc8cc0f1 not found: ID does not exist" Nov 02 07:23:01.006891 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.006798 6273 scope.go:122] "RemoveContainer" containerID="190305ce599cd3ddd29c6275ab803907f0a0a04a5520524df6fd82180e211a9d" Nov 02 07:23:01.149060 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.148972 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:23:01.195308 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.195281 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c0d3ab-dbbb6c947-w22lq"] Nov 02 07:23:01.252848 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.252824 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:23:01.256242 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:01.256219 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c0d3ab-5cfb9588b5-kcw9b"] Nov 02 07:23:02.178280 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:02.178236 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" path="/var/lib/kubelet/pods/a1bb5e72-b82f-4cc3-96e5-4a432ac38215/volumes" Nov 02 07:23:02.178787 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:02.178758 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52a2957-ccc4-4921-bec4-552d78bc029a" path="/var/lib/kubelet/pods/d52a2957-ccc4-4921-bec4-552d78bc029a/volumes" Nov 02 07:23:04.242456 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.242420 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:23:04.242898 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.242622 6273 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" podUID="a267274d-cb52-4f03-aafa-10cc4a597aae" containerName="maas-api" containerID="cri-o://de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250" gracePeriod=30 Nov 02 07:23:04.502158 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.502104 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:23:04.550603 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.550577 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") pod \"a267274d-cb52-4f03-aafa-10cc4a597aae\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " Nov 02 07:23:04.550725 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.550643 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\" (UniqueName: \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\") pod \"a267274d-cb52-4f03-aafa-10cc4a597aae\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " Nov 02 07:23:04.550776 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.550737 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\") pod \"a267274d-cb52-4f03-aafa-10cc4a597aae\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " Nov 02 07:23:04.550776 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.550765 6273 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\") pod \"a267274d-cb52-4f03-aafa-10cc4a597aae\" (UID: \"a267274d-cb52-4f03-aafa-10cc4a597aae\") " Nov 02 07:23:04.551146 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.551113 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca" pod "a267274d-cb52-4f03-aafa-10cc4a597aae" (UID: "a267274d-cb52-4f03-aafa-10cc4a597aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:23:04.552588 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.552560 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls" pod "a267274d-cb52-4f03-aafa-10cc4a597aae" (UID: "a267274d-cb52-4f03-aafa-10cc4a597aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:23:04.552706 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.552685 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p" pod "a267274d-cb52-4f03-aafa-10cc4a597aae" (UID: "a267274d-cb52-4f03-aafa-10cc4a597aae"). InnerVolumeSpecName "kube-api-access-xr28p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:04.552759 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.552741 6273 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs" pod "a267274d-cb52-4f03-aafa-10cc4a597aae" (UID: "a267274d-cb52-4f03-aafa-10cc4a597aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:23:04.652032 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.651974 6273 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-maas-api-tls\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:04.652032 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.652028 6273 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr28p\" (UniqueName: \"kubernetes.io/projected/a267274d-cb52-4f03-aafa-10cc4a597aae-kube-api-access-xr28p\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:04.652032 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.652039 6273 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a267274d-cb52-4f03-aafa-10cc4a597aae-openshift-service-ca\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:04.652326 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:04.652048 6273 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a267274d-cb52-4f03-aafa-10cc4a597aae-metrics-certs\") on node \"ip-10-0-142-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:05.021414 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.021383 6273 generic.go:356] "Generic (PLEG): container finished" podID="a267274d-cb52-4f03-aafa-10cc4a597aae" containerID="de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250" exitCode=0 Nov 02 07:23:05.021585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.021424 6273 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" Nov 02 07:23:05.021585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.021453 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" event={"ID":"a267274d-cb52-4f03-aafa-10cc4a597aae","Type":"ContainerDied","Data":"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250"} Nov 02 07:23:05.021585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.021496 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n" event={"ID":"a267274d-cb52-4f03-aafa-10cc4a597aae","Type":"ContainerDied","Data":"0654f594cd2cf762960d49e7d1cca83ef90adac8558c48fe9320e2deeed6b7e1"} Nov 02 07:23:05.021585 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.021520 6273 scope.go:122] "RemoveContainer" containerID="de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250" Nov 02 07:23:05.030767 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.030750 6273 scope.go:122] "RemoveContainer" containerID="de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250" Nov 02 07:23:05.031063 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:23:05.031045 6273 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250\": container with ID starting with de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250 not found: ID does not exist" containerID="de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250" Nov 02 07:23:05.031111 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.031078 6273 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250"} err="failed to get container status \"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250\": rpc error: code = NotFound desc = could not find container \"de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250\": container with ID starting with de3518ee5065f81e06d77800200fd2d02a937e176dd17f42f85bb6cd660da250 not found: ID does not exist" Nov 02 07:23:05.174315 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.174285 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:23:05.177467 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:05.177444 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c0d3ab-7f65d5dbd5-7wq2n"] Nov 02 07:23:06.178297 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:06.178268 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a267274d-cb52-4f03-aafa-10cc4a597aae" path="/var/lib/kubelet/pods/a267274d-cb52-4f03-aafa-10cc4a597aae/volumes" Nov 02 07:23:08.551460 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:23:08.551421 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:23:10.845179 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:10.845145 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:23:23.757705 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:23.757673 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:23:34.865718 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:34.865645 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:23:38.045189 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:38.045145 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh"] Nov 02 07:23:38.090395 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:38.090365 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wkrsh"] Nov 02 07:23:38.178007 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:38.177962 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b275d76-d355-4455-a112-cf44a7be3e3e" path="/var/lib/kubelet/pods/1b275d76-d355-4455-a112-cf44a7be3e3e/volumes" Nov 02 07:23:38.550528 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:23:38.550490 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:23:55.862006 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:23:55.861954 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:24:07.046924 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:07.046877 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm"] Nov 02 07:24:07.092984 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:07.092958 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z59gm"] Nov 02 07:24:08.178568 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:08.178538 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1887eb19-efa5-429e-bb20-992e7144f6f6" path="/var/lib/kubelet/pods/1887eb19-efa5-429e-bb20-992e7144f6f6/volumes" Nov 02 07:24:08.551259 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:24:08.551184 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:24:26.547432 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:26.547396 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n2zvg"] Nov 02 07:24:36.255838 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:36.255806 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85b9b944cd-5tx47_0bc2fcaa-2adb-4c3b-8300-46456515a594/manager/0.log" Nov 02 07:24:36.818017 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:36.817978 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8fqnv_97c0055c-9928-4129-a6f2-34ce3cf361f6/manager/0.log" Nov 02 07:24:37.572169 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:37.572138 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:24:38.058587 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:38.058559 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-67qww_5b5a208a-14d3-4fd1-adac-c6e4be0a27e9/server/0.log" Nov 02 07:24:38.273506 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:38.273477 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2b59c_53fcf074-8bd9-42be-8c55-c41317dfa9fa/manager/0.log" Nov 02 07:24:38.551918 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:24:38.551883 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:24:39.624123 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:39.624092 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbfc94f97-wfbt4_032eef7f-ff8a-4d63-b409-47270dbf4f17/maas-api/0.log" Nov 02 07:24:40.051471 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:40.051443 6273 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p"] Nov 02 07:24:40.053665 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:40.053639 6273 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdaa0dea-fbg4p"] Nov 02 07:24:40.178390 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:40.178362 6273 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5cb1ea-da82-4002-98f1-ab675e5451fb" path="/var/lib/kubelet/pods/cd5cb1ea-da82-4002-98f1-ab675e5451fb/volumes" Nov 02 07:24:42.916344 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:42.916309 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6ngfr_932c90b7-2eb1-415b-b2e1-98622b054566/kuadrant-console-plugin/0.log" Nov 02 07:24:43.604298 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:43.604269 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n2zvg_af82ea08-8e55-418c-a817-426f7e8721f1/limitador/0.log" Nov 02 07:24:45.118641 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:45.118610 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784594468b-7wznk_af32b8a2-f1c4-4adc-9883-de2d21a64c43/kube-auth-proxy/0.log" Nov 02 07:24:46.008261 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:46.008232 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6mt9r_7eef205e-a4bf-402c-83ce-f3bd42c0cf7c/istio-proxy/0.log" Nov 02 07:24:46.471896 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:46.471871 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-9qr85_3890b910-f647-455f-adb3-3d26ebbde75c/istio-proxy/0.log" Nov 02 07:24:47.644300 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:47.644273 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-w98qm_6b6c73eb-608b-4148-ba6d-0ccfff6306d6/istio-proxy/0.log" Nov 02 07:24:50.672273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:24:50.672243 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-wfdqg_ffe42dde-6598-48cf-9eb6-351ed0edb27b/main/0.log" Nov 02 07:25:01.035508 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:01.035474 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85b9b944cd-5tx47_0bc2fcaa-2adb-4c3b-8300-46456515a594/manager/0.log" Nov 02 07:25:01.695120 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:01.695088 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8fqnv_97c0055c-9928-4129-a6f2-34ce3cf361f6/manager/0.log" Nov 02 07:25:02.459091 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:02.459064 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:25:02.816629 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:02.816566 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-67qww_5b5a208a-14d3-4fd1-adac-c6e4be0a27e9/server/0.log" Nov 02 07:25:03.102420 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:03.102346 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2b59c_53fcf074-8bd9-42be-8c55-c41317dfa9fa/manager/0.log" Nov 02 07:25:04.277958 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:04.277927 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbfc94f97-wfbt4_032eef7f-ff8a-4d63-b409-47270dbf4f17/maas-api/0.log" Nov 02 07:25:07.170447 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:07.170420 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6ngfr_932c90b7-2eb1-415b-b2e1-98622b054566/kuadrant-console-plugin/0.log" Nov 02 07:25:07.706271 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:07.706248 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n2zvg_af82ea08-8e55-418c-a817-426f7e8721f1/limitador/0.log" Nov 02 07:25:08.552457 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:25:08.552425 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:25:09.104761 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:09.104723 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784594468b-7wznk_af32b8a2-f1c4-4adc-9883-de2d21a64c43/kube-auth-proxy/0.log" Nov 02 07:25:10.044066 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:10.044040 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6mt9r_7eef205e-a4bf-402c-83ce-f3bd42c0cf7c/istio-proxy/0.log" Nov 02 07:25:10.488913 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:10.488885 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-9qr85_3890b910-f647-455f-adb3-3d26ebbde75c/istio-proxy/0.log" Nov 02 07:25:11.686955 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:11.686926 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-w98qm_6b6c73eb-608b-4148-ba6d-0ccfff6306d6/istio-proxy/0.log" Nov 02 07:25:14.437722 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:14.437690 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-wfdqg_ffe42dde-6598-48cf-9eb6-351ed0edb27b/main/0.log" Nov 02 07:25:27.945556 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:27.945523 6273 scope.go:122] "RemoveContainer" containerID="8cfcf85a2c8c424691083db530a0ef3589f6849f2e56dbdb1674badf6295e275" Nov 02 07:25:27.953386 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:27.953363 6273 scope.go:122] "RemoveContainer" containerID="a37fe7a2667460366bf020d46e93c1f508b56a1da12ad7789651d89f48d119eb" Nov 02 07:25:38.552437 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:25:38.552387 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:25:43.877270 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:43.877236 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:25:44.441317 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:25:44.441287 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:26:08.553850 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:26:08.553812 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:26:24.402608 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:24.402578 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:26:24.971446 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:24.971416 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fb8fd585-tz5c5_a7695fda-4da8-4451-956c-aa9f9a780c69/manager/0.log" Nov 02 07:26:26.298016 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.297972 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-62jbh/must-gather-k4mzg"] Nov 02 07:26:26.298405 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298391 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" containerName="payload-pre-processing" Nov 02 07:26:26.298452 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298407 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" containerName="payload-pre-processing" Nov 02 07:26:26.298452 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298439 6273 cpu_manager.go:414] "containerMap: removing container" podUID="d52a2957-ccc4-4921-bec4-552d78bc029a" containerName="payload-processing" Nov 02 07:26:26.298452 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298444 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52a2957-ccc4-4921-bec4-552d78bc029a" containerName="payload-processing" Nov 02 07:26:26.298551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298462 6273 cpu_manager.go:414] "containerMap: removing container" podUID="a267274d-cb52-4f03-aafa-10cc4a597aae" containerName="maas-api" Nov 02 07:26:26.298551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298467 6273 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a267274d-cb52-4f03-aafa-10cc4a597aae" containerName="maas-api" Nov 02 07:26:26.298551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298534 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bb5e72-b82f-4cc3-96e5-4a432ac38215" containerName="payload-pre-processing" Nov 02 07:26:26.298551 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298548 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52a2957-ccc4-4921-bec4-552d78bc029a" containerName="payload-processing" Nov 02 07:26:26.298690 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.298555 6273 memory_manager.go:365] "RemoveStaleState removing state" podUID="a267274d-cb52-4f03-aafa-10cc4a597aae" containerName="maas-api" Nov 02 07:26:26.302022 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.301978 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.397707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.397685 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"kube-root-ca.crt\"" Nov 02 07:26:26.397707 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.397685 6273 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-62jbh\"/\"default-dockercfg-pqfzn\"" Nov 02 07:26:26.397869 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.397717 6273 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"openshift-service-ca.crt\"" Nov 02 07:26:26.399055 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.399036 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/must-gather-k4mzg"] Nov 02 07:26:26.430952 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.430913 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54a2f837-5059-441d-b1d4-bef6b75b5cc2-must-gather-output\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.431082 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.430971 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9wd\" (UniqueName: \"kubernetes.io/projected/54a2f837-5059-441d-b1d4-bef6b75b5cc2-kube-api-access-gx9wd\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.531839 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.531812 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54a2f837-5059-441d-b1d4-bef6b75b5cc2-must-gather-output\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.531966 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.531851 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9wd\" (UniqueName: \"kubernetes.io/projected/54a2f837-5059-441d-b1d4-bef6b75b5cc2-kube-api-access-gx9wd\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.532196 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.532177 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54a2f837-5059-441d-b1d4-bef6b75b5cc2-must-gather-output\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.587534 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.587477 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9wd\" (UniqueName: \"kubernetes.io/projected/54a2f837-5059-441d-b1d4-bef6b75b5cc2-kube-api-access-gx9wd\") pod \"must-gather-k4mzg\" (UID: \"54a2f837-5059-441d-b1d4-bef6b75b5cc2\") " pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.615673 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.615652 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/must-gather-k4mzg" Nov 02 07:26:26.978903 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.978822 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/must-gather-k4mzg"] Nov 02 07:26:26.981667 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:26:26.981641 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54a2f837_5059_441d_b1d4_bef6b75b5cc2.slice/crio-2222f339129ce45ad6cb8080e8b9f938e2a491c7ad442c6aa101fed4d7b630c8 WatchSource:0}: Error finding container 2222f339129ce45ad6cb8080e8b9f938e2a491c7ad442c6aa101fed4d7b630c8: Status 404 returned error can't find the container with id 2222f339129ce45ad6cb8080e8b9f938e2a491c7ad442c6aa101fed4d7b630c8 Nov 02 07:26:26.983680 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:26.983659 6273 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:26:27.006515 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:27.006481 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/must-gather-k4mzg" event={"ID":"54a2f837-5059-441d-b1d4-bef6b75b5cc2","Type":"ContainerStarted","Data":"2222f339129ce45ad6cb8080e8b9f938e2a491c7ad442c6aa101fed4d7b630c8"} Nov 02 07:26:27.974731 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:27.974697 6273 scope.go:122] "RemoveContainer" containerID="d7d45e32c4f8b1c616df3708acac9dfa9cd886d25d9f69cb80a105e8f83b0c63" Nov 02 07:26:29.022335 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:29.022293 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/must-gather-k4mzg" event={"ID":"54a2f837-5059-441d-b1d4-bef6b75b5cc2","Type":"ContainerStarted","Data":"06b18fb6f766be6638f255c5f63fad3aa39400be33ceef9831d0250c8a4580df"} Nov 02 07:26:29.022733 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:29.022344 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/must-gather-k4mzg" event={"ID":"54a2f837-5059-441d-b1d4-bef6b75b5cc2","Type":"ContainerStarted","Data":"bb65754b58b19eb89f7250b9254212c622aa122328091774ec87dea8843cfa7b"} Nov 02 07:26:29.130670 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:29.130620 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-62jbh/must-gather-k4mzg" podStartSLOduration=1.662571782 podStartE2EDuration="3.130607509s" totalImagesPullingTime="1.468035727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:26:29.128922374 +0000 UTC m=+4143.102257067" watchObservedRunningTime="2026-11-02 07:26:29.130607509 +0000 UTC m=+4143.103942192" Nov 02 07:26:35.940351 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:35.934567 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h5h5g_fab2c88a-127f-4391-870a-f9757c5f9b04/global-pull-secret-syncer/0.log" Nov 02 07:26:38.132587 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:38.129540 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gqnpb_853c435f-0786-450f-bdd9-f95cc551ba78/konnectivity-agent/0.log" Nov 02 07:26:38.563011 ip-10-0-142-103 kubenswrapper[6273]: E1102 07:26:38.562369 6273 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmlst" containerName="extractor" Nov 02 07:26:39.717353 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:39.717314 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-103.us-west-2.compute.internal_05718cfa86289f66de168b803f5f9362/haproxy/0.log" Nov 02 07:26:44.792124 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:44.792094 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6ngfr_932c90b7-2eb1-415b-b2e1-98622b054566/kuadrant-console-plugin/0.log" Nov 02 07:26:45.233486 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.233447 6273 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp"] Nov 02 07:26:45.242139 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.242101 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.259563 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.259526 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-lib-modules\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.259722 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.259645 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-podres\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.259722 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.259709 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-sys\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.259845 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.259824 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-proc\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.259902 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.259847 6273 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xbr\" (UniqueName: \"kubernetes.io/projected/8ef0416d-0317-4436-99df-e7ff3e534842-kube-api-access-q6xbr\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.322568 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.322530 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp"] Nov 02 07:26:45.360654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360622 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-proc\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360654 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360657 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xbr\" (UniqueName: \"kubernetes.io/projected/8ef0416d-0317-4436-99df-e7ff3e534842-kube-api-access-q6xbr\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360724 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-lib-modules\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360746 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-podres\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360752 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-proc\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360771 6273 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-sys\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360833 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-podres\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360842 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-lib-modules\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.360908 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.360861 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef0416d-0317-4436-99df-e7ff3e534842-sys\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.414152 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.414123 6273 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xbr\" (UniqueName: \"kubernetes.io/projected/8ef0416d-0317-4436-99df-e7ff3e534842-kube-api-access-q6xbr\") pod \"perf-node-gather-daemonset-dzjzp\" (UID: \"8ef0416d-0317-4436-99df-e7ff3e534842\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.560524 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.560456 6273 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:45.578533 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.578507 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n2zvg_af82ea08-8e55-418c-a817-426f7e8721f1/limitador/0.log" Nov 02 07:26:45.927473 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:45.927293 6273 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp"] Nov 02 07:26:45.930526 ip-10-0-142-103 kubenswrapper[6273]: W1102 07:26:45.930490 6273 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8ef0416d_0317_4436_99df_e7ff3e534842.slice/crio-a42ef62f85a4338854a3e14728c7acd2fd7ab458e941404d852ede74f64b9073 WatchSource:0}: Error finding container a42ef62f85a4338854a3e14728c7acd2fd7ab458e941404d852ede74f64b9073: Status 404 returned error can't find the container with id a42ef62f85a4338854a3e14728c7acd2fd7ab458e941404d852ede74f64b9073 Nov 02 07:26:46.171292 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:46.171244 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" event={"ID":"8ef0416d-0317-4436-99df-e7ff3e534842","Type":"ContainerStarted","Data":"68692a1eb442c7b90ad1200c706da0a64f733cb78d0003bb27778f83d0ccfca5"} Nov 02 07:26:46.171496 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:46.171310 6273 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" event={"ID":"8ef0416d-0317-4436-99df-e7ff3e534842","Type":"ContainerStarted","Data":"a42ef62f85a4338854a3e14728c7acd2fd7ab458e941404d852ede74f64b9073"} Nov 02 07:26:46.171496 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:46.171401 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:46.278097 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:46.277967 6273 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" podStartSLOduration=1.2779495060000001 podStartE2EDuration="1.277949506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:26:46.277513692 +0000 UTC m=+4160.250848377" watchObservedRunningTime="2026-11-02 07:26:46.277949506 +0000 UTC m=+4160.251284190" Nov 02 07:26:51.725813 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:51.725782 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7c2lz_b424524e-6361-4ccb-9de6-58c78d4021fa/kube-state-metrics/0.log" Nov 02 07:26:52.119780 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:52.119710 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7c2lz_b424524e-6361-4ccb-9de6-58c78d4021fa/kube-rbac-proxy-main/0.log" Nov 02 07:26:52.186809 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:52.186365 6273 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-dzjzp" Nov 02 07:26:52.514764 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:52.514737 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7c2lz_b424524e-6361-4ccb-9de6-58c78d4021fa/kube-rbac-proxy-self/0.log" Nov 02 07:26:52.915158 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:52.915130 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64f9df6957-68857_2ae6eb4c-062d-4118-9bc0-8ac9af0cc881/metrics-server/0.log" Nov 02 07:26:53.312621 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:53.312546 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-skv9z_7b70d77d-9aaa-44d6-8f1b-e68febbd921c/monitoring-plugin/0.log" Nov 02 07:26:54.938658 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:54.938625 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xg9n_20ebe6ec-1b67-42bd-8f17-873ea5c799a4/node-exporter/0.log" Nov 02 07:26:55.334251 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:55.334176 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xg9n_20ebe6ec-1b67-42bd-8f17-873ea5c799a4/kube-rbac-proxy/0.log" Nov 02 07:26:55.728636 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:55.728611 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xg9n_20ebe6ec-1b67-42bd-8f17-873ea5c799a4/init-textfile/0.log" Nov 02 07:26:58.518470 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:58.518443 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/prometheus/0.log" Nov 02 07:26:58.913662 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:58.913632 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/config-reloader/0.log" Nov 02 07:26:59.310231 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:59.310163 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/thanos-sidecar/0.log" Nov 02 07:26:59.707549 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:26:59.707522 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/kube-rbac-proxy-web/0.log" Nov 02 07:27:00.104726 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:00.104662 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/kube-rbac-proxy/0.log" Nov 02 07:27:00.505075 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:00.505046 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/kube-rbac-proxy-thanos/0.log" Nov 02 07:27:00.904951 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:00.904923 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9cbaba29-3141-44c5-ba74-b99b221a7762/init-config-reloader/0.log" Nov 02 07:27:01.309072 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:01.308982 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-shhdr_ba5c9a49-a662-4b53-bbe8-31fc85b64434/prometheus-operator/0.log" Nov 02 07:27:01.705462 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:01.705415 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-shhdr_ba5c9a49-a662-4b53-bbe8-31fc85b64434/kube-rbac-proxy/0.log" Nov 02 07:27:02.103401 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:02.103329 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7bxsc_72bc39c2-67ef-463a-ac5c-cc48dc9e00e8/prometheus-operator-admission-webhook/0.log" Nov 02 07:27:02.511462 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:02.511436 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75d975db85-8dlws_d6df5824-c856-4945-b1da-7027bf549f00/telemeter-client/0.log" Nov 02 07:27:02.908584 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:02.908561 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75d975db85-8dlws_d6df5824-c856-4945-b1da-7027bf549f00/reload/0.log" Nov 02 07:27:03.305273 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:03.305203 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75d975db85-8dlws_d6df5824-c856-4945-b1da-7027bf549f00/kube-rbac-proxy/0.log" Nov 02 07:27:03.712153 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:03.712123 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/thanos-query/0.log" Nov 02 07:27:04.108470 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:04.108391 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/kube-rbac-proxy-web/0.log" Nov 02 07:27:04.504883 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:04.504855 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/kube-rbac-proxy/0.log" Nov 02 07:27:04.936050 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:04.936022 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/prom-label-proxy/0.log" Nov 02 07:27:05.332638 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:05.332568 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/kube-rbac-proxy-rules/0.log" Nov 02 07:27:05.737051 ip-10-0-142-103 kubenswrapper[6273]: I1102 07:27:05.737020 6273 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5b54f4c7-hk8ml_206c88d3-cdff-496a-ac41-e8e7feca0ac8/kube-rbac-proxy-metrics/0.log"